Tool for Sequential Snapshotting of Aggregated Data from Streaming Data
https://doi.org/10.26907/1562-5419-2023-26-4-414-436
Abstract
n the modern world, streaming data has become widespread in many subject areas. The task of processing streaming data in real time, with minimal delay, is highly relevant.
In stream processing, data processing, various approximate algorithms are often used, which have much higher time and memory efficiency than exact algorithms. In addition, there is often a need to forecast the state of the stream.
Thus, there is currently a need for a tool for sequential snapshotting of aggregated data from streaming data, enabling flow state prediction and approximate algorithms for stream data processing.
The authors of the article have developed such a tool, reviewed its architecture and mechanism of functioning, and evaluated the prospects for its further development.
About the Authors
A. I. Gurianov
Kazan (Volga region) Federal University
Russian Federation
Russian Federation
A. S. Yakupov
Kazan (Volga region) Federal University
Russian Federation
Russian Federation
References
1. Гурьянова Э. А., Гурьянов А. И. Анализ и перспективы рынка SaaS в Российской Федерации // Вестник экономики, права и социологии. 2022. №1. С. 182–185.
2. Kolajo T., Daramola O., Adebiyi A. Big data stream analysis: a systematic literature review. // Journal of Big Data. 2019. Vol. 6.
3. https://doi.org/10.1186/s40537-019-0210-7
4. Маркова В. Д. Влияние цифровой экономики на бизнес // ЭКО. 2018. №12 (534). С. 7–22.
5. Определение потоковой передачи данных // Amazon Web Services (AWS). – URL: https://aws.amazon.com/ru/streaming-data/ (дата обращения 12.05.2023)
6. Ельченков Р. А., Дунаев М. Е., Зайцев К. С. Прогнозирование временных рядов при обработке потоковых данных в реальном времени // International Journal of Open Information Technologies. 2022. Т. 10, №6. С. 62–69.
7. Апатова Н. В. Управление в экосистеме бизнеса в период цифровой трансформации // Эффективное управление экономикой: проблемы и перспективы. 2022. С. 238–241.
8. Маркова В. Д., Кузнецова С. А. Развитие стратегического менеджмента в цифровой экономике // Вестник Томского государственного университета. Экономика. 2019. №48. С. 217–232. https://doi.org/10.17223/19988648/48/15
9. Петренко А. С., Петренко С. А. Технологии больших данных (big data) в области информационной безопасности // The 2018 Symposium on Cybersecurity of the Digital Economy. 2018. C. 248–255.
10. Трофимов В. В., Трофимова Л. А. О концепции управления на основе данных в условиях цифровой трансформации // Петербургский экономический журнал. 2021. №4. С. 149–155. https://doi.org/10.24412/2307-5368-2021-4-149-155
11. Логиновский О. В., Шестаков А. Л., Шинкарев А. А. Построение современных корпоративных информационных систем // Управление большими системами: сборник трудов. 2019. №81. С. 113–146.
12. https://doi.org/10.25728/ubs.2019.81.5
13. Alwaisi S. S. A., Abbood M. N., Jalil L. F., Kasim S., Fudzee M. F. M., Hadi R., Ismail M. A. A. Review on Big Data Stream Processing Applications: Contributions, Benefits, and Limitations // International Journal on Informatics Visualization. 2021. Vol. 5(4). P. 456–460. https://doi.org/10.30630/joiv.5.4.737
14. McSherry F. View Maintenance: A New Approach to Data Processing // Materialize Blog. 2020. URL: https://materialize.com/blog/olvm/ (дата обращения 12.05.2023)
15. Singh A., Garg S., Kaur R., Batra S., Kumar N., Zomaya A. Y. Probabilistic data structures for big data analytics: A comprehensive review // Knowledge-Based Systems. 2020. Vol. 188. https://doi.org/10.1016/j.knosys.2019.104987
16. Torres J. F., Hadjout D., Sebaa A., Martinez-Alvarez F., Troncoso A. Deep Learning for Time Series Forecasting: A Survey // Big Data. 2021. Vol 9(1). https://doi.org/10.1089/big.2020.0159
17. Brandt T. L., Grawunder M. Moving Object Stream Processing with Short-Time Prediction // Proceedings of the 8th ACM SIGSPATIAL Workshop on GeoStreaming. 2017. https://doi.org/10.1145/3148160.3148168
18. Incremental Computation in the Database // Materialize. – URL: https://materialize.com/guides/incremental-computation/ (дата обращения 12.05.2023)
19. Upserts in Differential Dataflow // Materialize Blog. 2020. URL: https://materialize.com/blog/upserts-in-differential-dataflow/ (дата обращения 12.05.2023)
20. artemgur/Diplom // GitHub. URL: https://github.com/artemgur/diplom (дата обращения 12.05.2023)
21. Materialize Documentation // Materialize. URL: https://materialize.com/docs/ (дата обращения 12.05.2023)
22. Data definition // ksqIDB Documentation. URL: https://docs.ksqldb.io/en/latest/reference/sql/data-definition/ (дата обращения 12.05.2023)
23. Streaming ingestion // Amazon Redshift. URL: https://docs.aws.amazon.com/redshift/latest/dg/materialized-view-streaming-ingestion.html (дата обращения 12.05.2023)
24. Confluent Community License Agreement // GitHub. 2018. URL: https://github.com/confluentinc/ksql/blob/master/LICENSE (дата обращения 12.05.2023)
25. Materialize Business Source License Agreement // GitHub. URL: https://github.com/MaterializeInc/materialize/blob/main/LICENSE (дата обращения 12.05.2023)
26. Ting D. Approximate Distinct Counts for Billions of Datasets // Proceedings of the 2019 International Conference on Management of Data. 2019. P. 69–86.
27. https://doi.org/10.1145/3299869.3319897
28. Fan L., Cao P., Almeida J., Broder A. Summary Cache: A Scalable Wide-Area Web Cache Sharing Protocol // IEEE/ACM Transactions on Networking. 2000. Vol. 8(3). P. 281–293. https://doi.org/10.1109/90.851975
29. Flajolet P., Fusy E., Gandouet O., Meunier F. HyperLogLog: the analysis of a near-optimal cardinality estimation algorithm // Discrete Mathematics & Theoretical Computer Science. 2007. P. 137–156.
30. Boyer R.S., Moore J.S. MJRTY – A Fast Majority Vote Algorithm // Automated Reasoning / ed. Boyer R. S. Dordrecht: Kluwer Academic Publishers, 1991. P. 105–117. https://doi.org/10.1007/978-94-011-3488-0_5
31. Singh B., Chaitra B. H. Comprehensive Review of Stream Processing Tools // International Research Journal of Engineering and Technology. 2020. Vol. 7(5). P. 3537–3540.
Review
For citations:
Gurianov A.I., Yakupov A.S. Tool for Sequential Snapshotting of Aggregated Data from Streaming Data. Russian Digital Libraries Journal. 2023;26(4):414-436. (In Russ.) https://doi.org/10.26907/1562-5419-2023-26-4-414-436
Views:
25
Email this article 