The Lie of the Average: How Class Incremental Learning Evaluation Deceives You?
This work addresses a critical evaluation flaw in CIL for researchers and practitioners, offering a more reliable method to assess model performance across diverse class sequences, though it is incremental as it builds on existing CIL frameworks.
The paper tackles the problem of biased evaluation in Class Incremental Learning (CIL) by showing that current protocols using small random samples fail to capture the full performance range, leading to underestimated variance and inaccurate mean estimates. It introduces EDGE, an evaluation protocol that adaptively identifies extreme sequences using inter-task similarity, resulting in more accurate distributional boundaries and actionable insights for model robustness.
Class Incremental Learning (CIL) requires models to continuously learn new classes without forgetting previously learned ones, while maintaining stable performance across all possible class sequences. In real-world settings, the order in which classes arrive is diverse and unpredictable, and model performance can vary substantially across different sequences. Yet mainstream evaluation protocols calculate mean and variance from only a small set of randomly sampled sequences. Our theoretical analysis and empirical results demonstrate that this sampling strategy fails to capture the full performance range, resulting in biased mean estimates and a severe underestimation of the true variance in the performance distribution. We therefore contend that a robust CIL evaluation protocol should accurately characterize and estimate the entire performance distribution. To this end, we introduce the concept of extreme sequences and provide theoretical justification for their crucial role in the reliable evaluation of CIL. Moreover, we observe a consistent positive correlation between inter-task similarity and model performance, a relation that can be leveraged to guide the search for extreme sequences. Building on these insights, we propose EDGE (Extreme case-based Distribution and Generalization Evaluation), an evaluation protocol that adaptively identifies and samples extreme class sequences using inter-task similarity, offering a closer approximation of the ground-truth performance distribution. Extensive experiments demonstrate that EDGE effectively captures performance extremes and yields more accurate estimates of distributional boundaries, providing actionable insights for model selection and robustness checking. Our code is available at https://github.com/AIGNLAI/EDGE.