DPASF: A Flink Library for Streaming Data preprocessing
This provides a practical tool for data scientists working with continuous Big Data streams, though it is incremental as it adapts existing algorithms to a streaming framework.
The authors tackled the lack of streaming data preprocessing tools for Big Data by developing DPASF, a library in Apache Flink that implements six popular algorithms for discretization and feature selection, showing it can reduce data size while maintaining or improving accuracy quickly.
Data preprocessing techniques are devoted to correct or alleviate errors in data. Discretization and feature selection are two of the most extended data preprocessing techniques. Although we can find many proposals for static Big Data preprocessing, there is little research devoted to the continuous Big Data problem. Apache Flink is a recent and novel Big Data framework, following the MapReduce paradigm, focused on distributed stream and batch data processing. In this paper we propose a data stream library for Big Data preprocessing, named DPASF, under Apache Flink. We have implemented six of the most popular data preprocessing algorithms, three for discretization and the rest for feature selection. The algorithms have been tested using two Big Data datasets. Experimental results show that preprocessing can not only reduce the size of the data, but to maintain or even improve the original accuracy in a short time. DPASF contains useful algorithms when dealing with Big Data data streams. The preprocessing algorithms included in the library are able to tackle Big Datasets efficiently and to correct imperfections in the data.