A comprehensive analysis of concept drift locality in data streams
This work addresses the problem of standardized evaluation for concept drift detectors in online learning, providing a comprehensive benchmark for researchers, though it is incremental in improving existing methodologies.
The paper tackled the challenge of concept drift in data streams by introducing a novel categorization based on drift locality and scale, and created a benchmark of 2,760 problems to evaluate 9 state-of-the-art detectors, revealing their strengths and weaknesses.
Adapting to drifting data streams is a significant challenge in online learning. Concept drift must be detected for effective model adaptation to evolving data properties. Concept drift can impact the data distribution entirely or partially, which makes it difficult for drift detectors to accurately identify the concept drift. Despite the numerous concept drift detectors in the literature, standardized procedures and benchmarks for comprehensive evaluation considering the locality of the drift are lacking. We present a novel categorization of concept drift based on its locality and scale. A systematic approach leads to a set of 2,760 benchmark problems, reflecting various difficulty levels following our proposed categorization. We conduct a comparative assessment of 9 state-of-the-art drift detectors across diverse difficulties, highlighting their strengths and weaknesses for future research. We examine how drift locality influences the classifier performance and propose strategies for different drift categories to minimize the recovery time. Lastly, we provide lessons learned and recommendations for future concept drift research. Our benchmark data streams and experiments are publicly available at https://github.com/gabrieljaguiar/locality-concept-drift.