An analysis of technological frameworks for data streams

TitleAn analysis of technological frameworks for data streams
Publication TypeJournal Article
Year of Publication2020
AuthorsPuentes, F., Pérez-Godoy M.D., González P., and del Jesus M. J.
JournalProgress in Artificial Intelligence
Volume9
Pagination239–261
Date Published06/2020
KeywordsBig Data, Big data frameworks, Data stream engines, Data streaming, Technological frameworks
Abstract

Real-time data analysis is becoming increasingly important in Big Data environments for addressing data stream issues. To this end, several technological frameworks have been developed, both open-source and proprietary, for the analysis of streaming data. This paper analyzes some open-source technological frameworks available for data streams, detailing their main characteristics. The objective is to facilitate decisions on which framework to use, meeting the needs of data mining methods for data streams. In this sense, there are important factors affecting the choice about which framework is most suitable for this purpose. Some of these factors are the existence of data mining libraries, the available documentation, the maturity of the platform, fault tolerance and processing guarantees, among others. Another decisive factor when choosing a data stream framework is its performance. For this reason, two comparisons have been made: a performance and latency comparison between Spark Streaming, Spark Structured Streaming, Storm, Flink and Samza following the Yahoo Streaming Benchmark methodology, and a comparison between Spark Streaming and Flink with a clustering algorithm for data streaming called streaming K-means.

DOI10.1007/s13748-020-00210-6