Drift-Aware Variational Autoencoder-based Anomaly Detection with Two-level Ensembling
This addresses the problem of detecting anomalies in unlabeled, nonstationary streaming data for domains like cybersecurity or IoT, though it is incremental as it builds on existing VAE and ensembling techniques.
The paper tackles anomaly detection in streaming data with concept drift by proposing VAE++ESDD, a method using incremental learning and two-level ensembling of VAEs and drift detectors, and it significantly outperforms strong baselines and state-of-the-art methods in experiments on real-world and synthetic datasets.
In today's digital world, the generation of vast amounts of streaming data in various domains has become ubiquitous. However, many of these data are unlabeled, making it challenging to identify events, particularly anomalies. This task becomes even more formidable in nonstationary environments where model performance can deteriorate over time due to concept drift. To address these challenges, this paper presents a novel method, VAE++ESDD, which employs incremental learning and two-level ensembling: an ensemble of Variational AutoEncoder(VAEs) for anomaly prediction, along with an ensemble of concept drift detectors. Each drift detector utilizes a statistical-based concept drift mechanism. To evaluate the effectiveness of VAE++ESDD, we conduct a comprehensive experimental study using real-world and synthetic datasets characterized by severely or extremely low anomalous rates and various drift characteristics. Our study reveals that the proposed method significantly outperforms both strong baselines and state-of-the-art methods.