ellibsЭлектронные библиотекиRussian Digital Libraries Journal1562-5419Казанский (Приволжский) федеральный университет10.26907/1562-5419-2025-28-5-1036-1056ellibs-607Research ArticleСтатьиНормализация текста, распознанного при помощи технологии оптического распознавания символов, с использованием легковесных LLMNormalization of Text Recognized by Optical Character Recognition using Lightweight LLMSВершининВладислав КонстантиновичVershininVladislav Konstantinovichvershinin@itmo.ruХодненкоИван ВладимировичKhodnenkoIvan VladimirovichIvan.Khodnenko@itmo.ruИвановСергей ВладимировичIvanovSergey Vladimirovichsvivanov@itmo.ruУниверситет ИТМОITMO University20251912202528510361056Copyright © Вершинин В.К., Ходненко И.В., Иванов С.В., 20252025Вершинин В.К., Ходненко И.В., Иванов С.В.Vershinin V.K., Khodnenko I.V., Ivanov S.V.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ellibs.elpub.ru/jour/article/view/607

Несмотря на значительный прогресс, технологии оптического распознавания символов (OCR) для исторических газет по-прежнему допускают 5–10% ошибок на уровне символов. В работе представлена полностью автоматизированная система нормализации пост-OCR, объединяющая легкие языковые модели (LLM) объемом 7–8 млрд параметров, обученные по инструкциям и квантизованные до 4 бит (INT4), с небольшим набором регулярных выражений. На наборе данных BLN600 (600 страниц британских газет XIX в.) лучшая модель YandexGPT-5-Instruct Q4 снижает Character Error Rate (CER) с 8.4% до 4.0% (–52.5%) и Word Error Rate (WER) с 20.2% до 6.5% (–67.8%), повышая при этом семантическое сходство до 0.962. Система работает на потребительском оборудовании (RTX-4060 Ti, 8 ГБ VRAM) со скоростью около 35 секунд на страницу и не требует дополнительного обучения или параллельных данных. Полученные результаты показывают, что компактные INT4-LLM являются практичной альтернативой крупным моделям для постобработки OCR исторических документов.

Despite recent progress, Optical Character Recognition (OCR) on historical newspapers still leaves 5–10% character errors. We present a fully automated post-OCR normalization pipeline that combines lightweight 7–8B instruction-tuned LLMs quantized to 4-bit (INT4) with a small set of regex rules. On the BLN600 benchmark (600 pages of 19th-century British newspapers), our best model YandexGPT-5-Instruct Q4 reduces Character Error Rate (CER) from 8.4% to 4.0% (–52.5%) and Word Error Rate (WER) from 20.2% to 6.5% (–67.8%), while raising semantic similarity to 0.962. The system runs on consumer hardware (RTX-4060 Ti, 8 GB VRAM) at about 35 seconds per page and requires no fine-tuning or parallel training data. These results indicate that compact INT4 LLMs are a practical alternative to large checkpoints for post-OCR cleanup of historical documents.

оптическое распознавание символовпост-OCR-коррекцияисторические газетыбольшие языковые моделиквантизацияINT4конвейер нормализацииошибка на уровне символовсемантическое сходстворегулярные выраженияYandexGPT-5легкие моделиобработка естественного языкацифровые гуманитарные наукиоцифровка документовoptical character recognitionpost-OCR correctionhistorical newspaperslarge language modelsquantizationINT4normalization pipelinecharacter error ratesemantic similarityregex rulesYandexGPT-5lightweight modelsnatural language processingdigital humanitiesdocument digitizationReferencesMemon J., Sami M., Khan R.A. Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR) // IEEE Access. 2020. Vol. 8. P. 142642–142668. https://doi.org/10.1109/ACCESS.2020.3012542Memon J., Sami M., Khan R.A. Handwritten Optical Character Recognition (OCR): A Comprehensive Systematic Literature Review (SLR) // IEEE Access. 2020. Vol. 8. P. 142642–142668. https://doi.org/10.1109/ACCESS.2020.3012542Thomas A., Gaizauskas R., Lu H. Leveraging LLMs for Post-OCR Correction of Historical Newspapers // Proceedings of the 2nd Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA). 2024. P. 116–121. https://doi.org/10.18653/v1/2024.lt4hala-1.6Thomas A., Gaizauskas R., Lu H. Leveraging LLMs for Post-OCR Correction of Historical Newspapers // Proceedings of the 2nd Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA). 2024. P. 116–121. https://doi.org/10.18653/v1/2024.lt4hala-1.6Bourne J. Scrambled text: training Language Models to correct OCR errors using synthetic data // arXiv preprint. 2024. arXiv:2409.19735. https://doi.org/10.48550/arXiv.2409.19735Bourne J. Scrambled text: training Language Models to correct OCR errors using synthetic data // arXiv preprint. 2024. arXiv:2409.19735. https://doi.org/10.48550/arXiv.2409.19735Holley R. How Good Can It Get? Analysing and Improving OCR Accuracy in Large-Scale Historic Newspaper Digitisation Programs // D-Lib Magazine. 2009. Vol. 15, No. 3/4. https://doi.org/10.1045/march2009-holleyHolley R. How Good Can It Get? Analysing and Improving OCR Accuracy in Large-Scale Historic Newspaper Digitisation Programs // D-Lib Magazine. 2009. Vol. 15, No. 3/4. https://doi.org/10.1045/march2009-holleyvan Strien D., Beelen K., Coll Ardanuy M., Hosseini K., McGillivray B., Tolfo G.S. Assessing the Impact of OCR Quality on Downstream NLP Tasks // Proceedings of the 12th International Conference on Agents and Artificial Intelligence (ICAART 2020). 2020. P. 484–496. https://doi.org/10.5220/0009169004840496van Strien D., Beelen K., Coll Ardanuy M., Hosseini K., McGillivray B., Tolfo G.S. Assessing the Impact of OCR Quality on Downstream NLP Tasks // Proceedings of the 12th International Conference on Agents and Artificial Intelligence (ICAART 2020). 2020. P. 484–496. https://doi.org/10.5220/0009169004840496Drobac S., Friberg Heppin K., Wirén M., Lindén K. Optical Character Recognition with Neural Networks and Post-Correction with Finite State Methods // International Journal on Document Analysis and Recognition (IJDAR). 2020. Vol. 23. P. 279–295. https://doi.org/10.1007/s10032-020-00359-9Drobac S., Friberg Heppin K., Wirén M., Lindén K. Optical Character Recognition with Neural Networks and Post-Correction with Finite State Methods // International Journal on Document Analysis and Recognition (IJDAR). 2020. Vol. 23. P. 279–295. https://doi.org/10.1007/s10032-020-00359-9Neudecker C., Antonacopoulos A. Making Europe’s Historical Newspapers Searchable // DAS 2016 Workshop / Europeana Newspapers. 2016.(Workshop paper). URL: https://www.primaresearch.org/www/files/das2016/Europeana%20Newspapers.pdf.Neudecker C., Antonacopoulos A. Making Europe’s Historical Newspapers Searchable // DAS 2016 Workshop / Europeana Newspapers. 2016.(Workshop paper). URL: https://www.primaresearch.org/www/files/das2016/Europeana%20Newspapers.pdf.Boillet M., Kermorvant C., Paquet T. Robust text line detection in historical documents: learning and evaluation methods // International Journal on Document Analysis and Recognition (IJDAR). 2022. Vol. 25. P. 95–114. https://doi.org/10.1007/s10032-022-00395-7Boillet M., Kermorvant C., Paquet T. Robust text line detection in historical documents: learning and evaluation methods // International Journal on Document Analysis and Recognition (IJDAR). 2022. Vol. 25. P. 95–114. https://doi.org/10.1007/s10032-022-00395-7Ermakova L., Tolfo G.S., Hosseini K. On the Impact of OCR Quality on Named Entity Extraction from Historical Newspapers // DH Benelux 2021 (Extended abstracts). 2021. URL: https://dhbenelux.org/wp-content/uploads/booklet2021.pdf#page=66Ermakova L., Tolfo G.S., Hosseini K. On the Impact of OCR Quality on Named Entity Extraction from Historical Newspapers // DH Benelux 2021 (Extended abstracts). 2021. URL: https://dhbenelux.org/wp-content/uploads/booklet2021.pdf#page=66Kettunen K. Optical Character Recognition Quality Affects Perceived Usefulness and Trust // arXiv preprint. 2022. arXiv:2209.08222.Kettunen K. Optical Character Recognition Quality Affects Perceived Usefulness and Trust // arXiv preprint. 2022. arXiv:2209.08222.Sreelekha S., Sumam A.R., Nair R.R. Systematic Review on Text Normalization Techniques and Its Approach to Non-Standard Words // Preprint. 2023 (ResearchGate). URL: https://www.researchgate.net/publication/373877004.Sreelekha S., Sumam A.R., Nair R.R. Systematic Review on Text Normalization Techniques and Its Approach to Non-Standard Words // Preprint. 2023 (ResearchGate). URL: https://www.researchgate.net/publication/373877004.Shi Y., Peng D., Liao W., Lin Z., Chen X., Liu C., Zhang Y., Jin L. Exploring OCR Capabilities of GPT-4V(ision): A Quantitative and In-Depth Evaluation // arXiv preprint. 2023. arXiv:2310.16809.Shi Y., Peng D., Liao W., Lin Z., Chen X., Liu C., Zhang Y., Jin L. Exploring OCR Capabilities of GPT-4V(ision): A Quantitative and In-Depth Evaluation // arXiv preprint. 2023. arXiv:2310.16809.Guan S., Xu C., Lin M., Greene D. Effective Synthetic Data and Test-Time Adaptation for OCR Correction // Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024). 2024. P. 15412–15425 (ACL Anthology).Guan S., Xu C., Lin M., Greene D. Effective Synthetic Data and Test-Time Adaptation for OCR Correction // Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024). 2024. P. 15412–15425 (ACL Anthology).Kanerva J., Ledins C., Käpyaho S., Ginter F. OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches // Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025). 2025. Tallinn, Estonia (ACL Anthology).Kanerva J., Ledins C., Käpyaho S., Ginter F. OCR Error Post-Correction with LLMs in Historical Documents: No Free Lunches // Proceedings of the Third Workshop on Resources and Representations for Under-Resourced Languages and Domains (RESOURCEFUL-2025). 2025. Tallinn, Estonia (ACL Anthology).Rijhwani S., Anastasopoulos A., Neubig G. OCR Post-Correction for Endangered Language Texts // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. P. 5931–5942. https://doi.org/10.18653/v1/2020.emnlp-main.478Rijhwani S., Anastasopoulos A., Neubig G. OCR Post-Correction for Endangered Language Texts // Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. P. 5931–5942. https://doi.org/10.18653/v1/2020.emnlp-main.478Jin R., Du J., Huang W., Liu W., Luan J., Wang B., Xiong D. A Comprehensive Evaluation of Quantization Strategies for Large Language Models // Findings of ACL 2024 (also arXiv preprint). 2024. arXiv:2402.16775. https://doi.org/10.48550/arXiv.2402.16775Jin R., Du J., Huang W., Liu W., Luan J., Wang B., Xiong D. A Comprehensive Evaluation of Quantization Strategies for Large Language Models // Findings of ACL 2024 (also arXiv preprint). 2024. arXiv:2402.16775. https://doi.org/10.48550/arXiv.2402.16775Mekala A., Atmakuru A., Song Y., Karpinska M., Iyyer M. Does Quantization Affect Models’ Performance on Long-Context Tasks? // arXiv preprint. 2025. arXiv:2505.20276. https://doi.org/10.48550/arXiv.2505.20276.Mekala A., Atmakuru A., Song Y., Karpinska M., Iyyer M. Does Quantization Affect Models’ Performance on Long-Context Tasks? // arXiv preprint. 2025. arXiv:2505.20276. https://doi.org/10.48550/arXiv.2505.20276.Booth C.W., Thomas A., Gaizauskas R. BLN600: A Parallel Corpus of Machine/Human Transcribed Nineteenth-Century Newspaper Texts // Proceedings of LREC-COLING 2024. 2024. P. 2440–2446. https://doi.org/10.15131/shef.data.25439023.Booth C.W., Thomas A., Gaizauskas R. BLN600: A Parallel Corpus of Machine/Human Transcribed Nineteenth-Century Newspaper Texts // Proceedings of LREC-COLING 2024. 2024. P. 2440–2446. https://doi.org/10.15131/shef.data.25439023.Gupta H., Del Corro L., Broscheit S., Hoffart J., Brenner E. Unsupervised Multi-View Post-OCR Error Correction with Language Models // Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2021. P. 8647–8652. https://doi.org/10.18653/v1/2021.emnlp-main.680Gupta H., Del Corro L., Broscheit S., Hoffart J., Brenner E. Unsupervised Multi-View Post-OCR Error Correction with Language Models // Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2021. P. 8647–8652. https://doi.org/10.18653/v1/2021.emnlp-main.680

The authors declare that there are no conflicts of interest present.