A System for Testing Controllers Based on On-Screen Text Recognition
https://doi.org/10.26907/1562-5419-2025-28-6-1368-1384
Abstract
A solution for the problem of testing controllers based on reading information from their screens is described. A hardware and software system has been developed for this purpose, consisting of a camera and software modules implementing the necessary algorithms and methods: an image preprocessing module; a menu type detection module; a font character processing module; a text reading module, including one written in various fonts; and the testing module itself. The system has been developed for a specific type of controller with a monochrome 128x64 pixel display. All methods are implemented in Python using popular libraries. The system has been launched into test operation and currently automates several of the most labor-intensive tests. The test set can be expanded using plugins.
About the Author
Aleksandr Aleksandrovich Dokukin
Federal Research Center “Computer Science and Control” of the Russian Academy of Sciences
Russian Federation
Russian Federation
References
1. SmartWEB-K (In Russian) // [Electronic resource] https://www.teplostart.ru/download/booklets/SmartWEB-K.pdf (access date: 06.10.2025)
2. Dokukin A.A. O postroenii sistemy schityvaniya informacii s ekrana kontrollera (in Russian) // Intellectual Data Processing (IDP-2024): Book of Abstracts 15th Int. Conf. (Grodno, September 23–27, 2024) 2024. P. 92–93.
3. Dokukin A.A. Razvitie sistemy schityvaniya informacii s ekrana kontrollera (In Russian) // Mathematical Methods for Pattern Recognition (MMPR-2025): Book of Abstracts 22nd Russian Conf. with Int. Participation (Murom, September 22–26, 2025) 2025. P. 139–140.
4. Le Cun Y., Boser B., Denker J. S., Henderson D., Howard R. E., Hubbard W., Jackel L. D. Handwritten Digit Recognition with a Back-Propagation Network // NIPS'89: Proc. 2nd Int. Conf. Neural Information Processing Systems. 1989. P. 396–404. https://proceedings.neurips.cc/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf
5. Ansel J., Yang E., He H. et al. PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation // 29th ACM Int. Conf. Architectural Support for Programming Languages and Operating Systems (ASPLOS '24). 2024. Vol. 2. https://doi.org/10.1145/3620665.3640366
6. Otsu N. A Threshold Selection Method from Gray-Level Histograms // IEEE Transact. on Systems, Man, and Cybernetics 9.1. 1979. P. 62–66. https://doi.org/10.1109/ TSMC.1979.4310076
7. Bradski G. The OpenCV Library // Dr. Dobb's J. Software Tools. 2000. Vol. 120. P. 122–125.
Review
For citations:
Dokukin A.A. A System for Testing Controllers Based on On-Screen Text Recognition. Russian Digital Libraries Journal. 2025;28(6):1368-1384. (In Russ.) https://doi.org/10.26907/1562-5419-2025-28-6-1368-1384
Views:
21
JATS XML
Email this article 