Sampling Streaming Data with Parallel Vector Quantization -- PVQ
This addresses class imbalance challenges in streaming data for enterprise cloud applications, though it is incremental as it builds on existing vector quantization and sampling techniques.
The paper tackles class imbalance in streaming data by introducing a vector quantization-based sampling method, which improves classification accuracy across multiple ML models, including Multilayered Perceptron, Support Vector Machines, K-Nearest Neighbour, and Random Forests, as demonstrated on network traffic and anomaly datasets.
Accumulation of corporate data in the cloud has attracted more enterprise applications to the cloud creating data gravity. As a consequence, network traffic has become more cloud centric. This increase in cloud centric traffic poses new challenges in designing learning systems for streaming data due to class imbalance. The number of classes plays a vital role in the accuracy of the classifiers built from the data streams. In this paper, we present a vector quantization-based sampling method, which substantially reduces the class imbalance in data streams. We demonstrate its effectiveness by conducting experiments on network traffic and anomaly dataset with commonly used ML model building methods; Multilayered Perceptron on TensorFlow backend, Support Vector Machines, K-Nearest Neighbour, and Random Forests. We built models using parallel processing, batch processing, and randomly selecting samples. We show that the accuracy of classification models improves when the data streams are pre-processed with our method. We used out of the box hyper-parameters of these classifiers and auto sklearn for hyperparameter optimization.