ellibsЭлектронные библиотекиRussian Digital Libraries Journal1562-5419Казанский (Приволжский) федеральный университет10.26907/1562-5419-2022-25-3-198-245ellibs-328Research ArticleСтатьиПрограммный фреймворк для реализации взаимодействия с пользовательскими интерфейсами iOS-приложений на основе окулографииSoftware Framework for Implementing User Interface Interaction in IOS Applications Based on OculographyАфанасьевН. С.AfanasevN. S.reqenoxus@gmail.comКазанский (Приволжский) Федеральный университетKazan (Volga region) Federal University202228062022253198245Copyright © Афанасьев Н.С., 20222022Афанасьев Н.С.Afanasev N.S.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ellibs.elpub.ru/jour/article/view/328

Использование технологий отслеживания взгляда для взаимодействия с пользовательским интерфейсом iOS-приложений существенно затруднено отсутствием унифицированного подхода к их интеграции. Существующие решения либо жестко ограничены своей предметной областью, либо представляют собой исключительно исследовательские проекты, непригодные для решения прикладных задач. В статье рассматривается создание фреймворка, осуществляющего отслеживание взгляда пользователя на экране Apple-устройств с использованием нативных технологий, а также предоставляющего унифицированный подход к разработке приложений, управляемых при помощи взгляда.

Usage of gaze tracking technologies for the purpose of user interface interaction in iOS applications is significantly hampered by the absence of a unified approach to their integration. Current solutions are either strictly limited to their own use-case or made solely for research purposes and thus inapplicable to real-world problems. The focus of this article is the development of a software framework that performs gaze tracking using native technologies and suggests a unified approach to the development of gaze-driven iOS applications.

отслеживание взглядаокулографияобработка жестовgaze trackingeye trackingTrueDepthARKitSceneKitUIKitiOSUXUIReferencesEsiyok C. et al. Novel hands-free interaction techniques based on the software switch approach for computer access with head movements // Universal Access in the Information Society. 2020. P. 1–15. https://doi.org/10.1007/s10209-020-00748-1Esiyok C. et al. Novel hands-free interaction techniques based on the software switch approach for computer access with head movements // Universal Access in the Information Society. 2020. P. 1–15. https://doi.org/10.1007/s10209-020-00748-1Roig-Maimó M.F. et al. Evaluation of a mobile head-tracker interface for accessibility // International Conference on Computers Helping People with Special Needs. Springer, Cham, 2016. P. 449–456. https://doi.org/10.1007/978-3-319-41267-2_63Roig-Maimó M.F. et al. Evaluation of a mobile head-tracker interface for accessibility // International Conference on Computers Helping People with Special Needs. Springer, Cham, 2016. P. 449–456. https://doi.org/10.1007/978-3-319-41267-2_63Abbaszadegan M., Yaghoubi S., MacKenzie I.S. TrackMaze: A comparison of head-tracking, eye-tracking, and tilt as input methods for mobile games // International Conference on Human-Computer Interaction. Springer, Cham, 2018. P. 393–405. https://doi.org/10.1007/978-3-319-91250-9_31Abbaszadegan M., Yaghoubi S., MacKenzie I.S. TrackMaze: A comparison of head-tracking, eye-tracking, and tilt as input methods for mobile games // International Conference on Human-Computer Interaction. Springer, Cham, 2018. P. 393–405. https://doi.org/10.1007/978-3-319-91250-9_31Tupikovskaja-Omovie Z., Tyler D. Clustering consumers' shopping journeys: eye tracking fashion m-retail // Journal of Fashion Marketing and Management: An International Journal. 2020. Т. 24. №. 3. P. 381–398. https://doi.org/10.1108/JFMM-09-2019-0195Tupikovskaja-Omovie Z., Tyler D. Clustering consumers' shopping journeys: eye tracking fashion m-retail // Journal of Fashion Marketing and Management: An International Journal. 2020. Т. 24. №. 3. P. 381–398. https://doi.org/10.1108/JFMM-09-2019-0195Garbutt M. et al. The embodied gaze: Exploring applications for mobile eye tracking in the art museum // Visitor Studies. 2020. Vol. 23. No. 1. P. 82–100. https://doi.org/10.1080/10645578.2020.1750271Garbutt M. et al. The embodied gaze: Exploring applications for mobile eye tracking in the art museum // Visitor Studies. 2020. Vol. 23. No. 1. P. 82–100. https://doi.org/10.1080/10645578.2020.1750271Vogt M., Rips A., Emmelmann C. Comparison of iPad Pro®’s LiDAR and TrueDepth capabilities with an industrial 3D scanning solution // Technologies. 2021. Vol. 9. No. 2. P. 25. https://doi.org/10.3390/technologies9020025Vogt M., Rips A., Emmelmann C. Comparison of iPad Pro®’s LiDAR and TrueDepth capabilities with an industrial 3D scanning solution // Technologies. 2021. Vol. 9. No. 2. P. 25. https://doi.org/10.3390/technologies9020025Breitbarth A. et al. Measurement accuracy and dependence on external influences of the iPhone X TrueDepth sensor // Photonics and Education in Measurement Science 2019. International Society for Optics and Photonics, 2019. Vol. 11144. P. 1114407. https://doi.org/10.1117/12.2530544Breitbarth A. et al. Measurement accuracy and dependence on external influences of the iPhone X TrueDepth sensor // Photonics and Education in Measurement Science 2019. International Society for Optics and Photonics, 2019. Vol. 11144. P. 1114407. https://doi.org/10.1117/12.2530544Number of smartphone users worldwide from 2016 to 2023 // Statista – The Statistics Portal for Market data, Market Research and Market Studies. URL: https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/Number of smartphone users worldwide from 2016 to 2023 // Statista – The Statistics Portal for Market data, Market Research and Market Studies. URL: https://www.statista.com/statistics/330695/number-of-smartphone-users-worldwide/Krafka K. et al. Eye tracking for everyone // Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition. 2016. P. 2176–2184. https://doi.org/10.1109/CVPR.2016.239Krafka K. et al. Eye tracking for everyone // Proceedings of the IEEE Conference On Computer Vision And Pattern Recognition. 2016. P. 2176–2184. https://doi.org/10.1109/CVPR.2016.239Huang M. X. et al. Screenglint: Practical, in-situ gaze estimation on smartphones // Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017. P. 2546–2557. https://doi.org/10.1145/3025453.3025794Huang M. X. et al. Screenglint: Practical, in-situ gaze estimation on smartphones // Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2017. P. 2546–2557. https://doi.org/10.1145/3025453.3025794Brousseau B., Rose J., Eizenman M. Smarteye: An accurate infrared eye tracking system for smartphones // 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON). IEEE, 2018. P. 951–959. https://doi.org/10.1109/UEMCON.2018.8796799Brousseau B., Rose J., Eizenman M. Smarteye: An accurate infrared eye tracking system for smartphones // 2018 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON). IEEE, 2018. P. 951–959. https://doi.org/10.1109/UEMCON.2018.8796799Hawkeye Access | Control your iOS device using your eyes. URL: https://www.usehawkeye.com/accessibilityHawkeye Access | Control your iOS device using your eyes. URL: https://www.usehawkeye.com/accessibilityKong A. et al. EyeMU Interactions: Gaze+ IMU Gestures on Mobile Devices // Proceedings of the 2021 International Conference on Multimodal Interaction. 2021. P. 577–585. https://doi.org/10.1145/3462244.3479938Kong A. et al. EyeMU Interactions: Gaze+ IMU Gestures on Mobile Devices // Proceedings of the 2021 International Conference on Multimodal Interaction. 2021. P. 577–585. https://doi.org/10.1145/3462244.3479938Skyle 2 for iPad – eyeV. URL: https://eyev.de/en/ipad/Skyle 2 for iPad – eyeV. URL: https://eyev.de/en/ipad/Cicek M. et al. Mobile head tracking for ecommerce and beyond //Electronic Imaging. 2020. Vol. 2020. No. 3. P. 303-1–303-12. https://doi.org/10.48550/arXiv.1812.07143Cicek M. et al. Mobile head tracking for ecommerce and beyond //Electronic Imaging. 2020. Vol. 2020. No. 3. P. 303-1–303-12. https://doi.org/10.48550/arXiv.1812.07143Kaufman A.E., Bandopadhay A., Shaviv B.D. An eye tracking computer user interface // Proceedings of 1993 IEEE Research Properties in Virtual Reality Symposium. IEEE, 1993. P. 120–121. https://doi.org/10.1109/VRAIS.1993.378254Kaufman A.E., Bandopadhay A., Shaviv B.D. An eye tracking computer user interface // Proceedings of 1993 IEEE Research Properties in Virtual Reality Symposium. IEEE, 1993. P. 120–121. https://doi.org/10.1109/VRAIS.1993.378254Gibaldi A. et al. Evaluation of the Tobii EyeX Eye tracking controller and Matlab toolkit for research // Behavior Research Methods. 2017. Vol. 49. No. 3. P. 923–946. https://doi.org/10.3758/s13428-016-0762-9Gibaldi A. et al. Evaluation of the Tobii EyeX Eye tracking controller and Matlab toolkit for research // Behavior Research Methods. 2017. Vol. 49. No. 3. P. 923–946. https://doi.org/10.3758/s13428-016-0762-9Xu P. et al. Turkergaze: Crowdsourcing saliency with webcam based eye tracking //arXiv preprint arXiv:1504.06755. 2015. https://doi.org/10.48550/arXiv.1504.06755Xu P. et al. Turkergaze: Crowdsourcing saliency with webcam based eye tracking //arXiv preprint arXiv:1504.06755. 2015. https://doi.org/10.48550/arXiv.1504.06755Qiao X. et al. A new era for web AR with mobile edge computing // IEEE Internet Computing. 2018. Vol. 22. No. 4. P. 46–55. https://doi.org/10.1109/MIC.2018.043051464Qiao X. et al. A new era for web AR with mobile edge computing // IEEE Internet Computing. 2018. Vol. 22. No. 4. P. 46–55. https://doi.org/10.1109/MIC.2018.043051464Taaban R.A., Croock M.S., Korial A.E. Eye Tracking Based Mobile Application // International Journal of Advanced Research in Computer Engineering & Technology (IJARCET). 2018. Vol. 7. No. 3. P. 246–250.Taaban R.A., Croock M.S., Korial A.E. Eye Tracking Based Mobile Application // International Journal of Advanced Research in Computer Engineering & Technology (IJARCET). 2018. Vol. 7. No. 3. P. 246–250.Heryadi Y. et al. Mata: An Android Eye-Tracking Based User Interface Control Application // Journal of Games, Game Art, and Gamification. 2016. Vol. 1. No. 1. P. 35–40. https://doi.org/10.21512/jggag.v1i1.7249Heryadi Y. et al. Mata: An Android Eye-Tracking Based User Interface Control Application // Journal of Games, Game Art, and Gamification. 2016. Vol. 1. No. 1. P. 35–40. https://doi.org/10.21512/jggag.v1i1.7249Greinacher R., Voigt-Antons J.N. Accuracy Assessment of ARKit 2 Based Gaze Estimation // International Conference on Human-Computer Interaction. Springer, Cham, 2020. P. 439–449. https://doi.org/10.1007/978-3-030-49059-1_32Greinacher R., Voigt-Antons J.N. Accuracy Assessment of ARKit 2 Based Gaze Estimation // International Conference on Human-Computer Interaction. Springer, Cham, 2020. P. 439–449. https://doi.org/10.1007/978-3-030-49059-1_32devicekit/DeviceKit: DeviceKit is a value-type replacement of UIDevice. URL: https://github.com/devicekit/DeviceKitdevicekit/DeviceKit: DeviceKit is a value-type replacement of UIDevice. URL: https://github.com/devicekit/DeviceKitblendShapes | Apple Developer Documentation. URL: https://developer.apple.com/documentation/arkit/arfaceanchor/2928251-blendshapesblendShapes | Apple Developer Documentation. URL: https://developer.apple.com/documentation/arkit/arfaceanchor/2928251-blendshapesinit(target:action:) | Apple Developer Documentation. URL: https://developer.apple.com/documentation/uikit/uigesturerecognizer/1624211-initinit(target:action:) | Apple Developer Documentation. URL: https://developer.apple.com/documentation/uikit/uigesturerecognizer/1624211-initSwift.org – Package Manager. URL: https://www.swift.org/package-manager/Swift.org – Package Manager. URL: https://www.swift.org/package-manager/Optimizing ProMotion Refresh Rates for iPhone 13 Pro and iPad Pro | Apple Developer Documentation. URL: https://developer.apple.com/library/archive/technotes/tn2460/_index.htmlOptimizing ProMotion Refresh Rates for iPhone 13 Pro and iPad Pro | Apple Developer Documentation. URL: https://developer.apple.com/library/archive/technotes/tn2460/_index.htmlGitHub – ReQEnoxus/gaze-tracker: UIGestureRecognizer extension based on GazeTracking. URL: https://github.com/ReQEnoxus/gaze-trackerGitHub – ReQEnoxus/gaze-tracker: UIGestureRecognizer extension based on GazeTracking. URL: https://github.com/ReQEnoxus/gaze-tracker

The authors declare that there are no conflicts of interest present.