Dark Experience for General Continual Learning: a Strong, Simple Baseline
This addresses the practical scenario in continual learning where task boundaries are blurred and distributions shift, offering a strong baseline for researchers, though it is incremental as it builds on existing techniques like rehearsal and distillation.
The paper tackles the problem of General Continual Learning (GCL) where data streams lack clear task boundaries and offline training is not feasible, by proposing Dark Experience Replay, a simple baseline that mixes rehearsal with knowledge distillation and regularization to match network logits for consistency, and shows it outperforms existing approaches on benchmarks like MNIST-360 while using limited resources.
Continual Learning has inspired a plethora of approaches and evaluation settings; however, the majority of them overlooks the properties of a practical scenario, where the data stream cannot be shaped as a sequence of tasks and offline training is not viable. We work towards General Continual Learning (GCL), where task boundaries blur and the domain and class distributions shift either gradually or suddenly. We address it through mixing rehearsal with knowledge distillation and regularization; our simple baseline, Dark Experience Replay, matches the network's logits sampled throughout the optimization trajectory, thus promoting consistency with its past. By conducting an extensive analysis on both standard benchmarks and a novel GCL evaluation setting (MNIST-360), we show that such a seemingly simple baseline outperforms consolidated approaches and leverages limited resources. We further explore the generalization capabilities of our objective, showing its regularization being beneficial beyond mere performance.