How well does Classification Accuracy capture Concept Drift Detection Quality? An overview of Concept Drift Detection evaluation
This work addresses the lack of a unified evaluation framework for concept drift detection methods, which is a problem for researchers and practitioners working with nonstationary data streams.
This paper investigates the relationship between classification accuracy and eight concept drift detection quality metrics across seven synthetic data stream generation tools. The study aims to identify the most informative set of metrics for evaluating concept drift detection methods.
Data streams are nowadays among the most frequently analyzed data structures, with the concept drift posing a major challenge encountered by processing systems. Despite the proposition of numerous solutions to counteract the accuracy degeneration due to concept drift, the scientific community has not yet established a unified framework for evaluating the concept drift detection task. Existing research often relies on classification quality metrics, but these can be affected by multiple factors and may not reliably reflect drift detection quality. In this work, we present an in-depth overview of the relationship between metrics for quantifying drift detection quality and classification performance in synthetic nonstationary data streams. The proposed research studies eight drift detection quality metrics in relation to the classifier's performance across seven synthetic data stream generation tools, additionally considering drift dynamics as a factor. The studies aim to identify the most informative set of drift detection quality metrics and provide a deep understanding of the method's evaluation.