CLDyB: Towards Dynamic Benchmarking for Continual Learning with Pre-trained Models
This work addresses the need for more realistic and rigorous evaluation in continual learning, particularly for researchers developing methods with pre-trained models, though it is incremental as it focuses on benchmarking rather than a new learning algorithm.
The authors tackled the problem of evaluating continual learning methods with pre-trained models by introducing CLDyB, a dynamic benchmarking framework that identifies challenging task sequences, leading to the discovery of weaknesses in state-of-the-art methods where performance drops significantly.
The advent of the foundation model era has sparked significant research interest in leveraging pre-trained representations for continual learning (CL), yielding a series of top-performing CL methods on standard evaluation benchmarks. Nonetheless, there are growing concerns regarding potential data contamination during the pre-training stage. Furthermore, standard evaluation benchmarks, which are typically static, fail to capture the complexities of real-world CL scenarios, resulting in saturated performance. To address these issues, we describe CL on dynamic benchmarks (CLDyB), a general computational framework based on Markov decision processes for evaluating CL methods reliably. CLDyB dynamically identifies inherently difficult and algorithm-dependent tasks for the given CL methods, and determines challenging task orders using Monte Carlo tree search. Leveraging CLDyB, we first conduct a joint evaluation of multiple state-of-the-art CL methods, leading to a set of commonly challenging and generalizable task sequences where existing CL methods tend to perform poorly. We then conduct separate evaluations of individual CL methods using CLDyB, discovering their respective strengths and weaknesses. The source code and generated task sequences are publicly accessible at https://github.com/szc12153/CLDyB.