Exemplar-free Continual Representation Learning via Learnable Drift Compensation
This addresses the challenge of forgetting in continual learning for AI systems that need to adapt over time without storing old data, though it is incremental as it builds on prototype-based methods.
The paper tackles the problem of semantic drift in exemplar-free continual representation learning by proposing Learnable Drift Compensation (LDC), which effectively mitigates drift and achieves state-of-the-art performance in supervised and semi-supervised settings across multiple datasets.
Exemplar-free class-incremental learning using a backbone trained from scratch and starting from a small first task presents a significant challenge for continual representation learning. Prototype-based approaches, when continually updated, face the critical issue of semantic drift due to which the old class prototypes drift to different positions in the new feature space. Through an analysis of prototype-based continual learning, we show that forgetting is not due to diminished discriminative power of the feature extractor, and can potentially be corrected by drift compensation. To address this, we propose Learnable Drift Compensation (LDC), which can effectively mitigate drift in any moving backbone, whether supervised or unsupervised. LDC is fast and straightforward to integrate on top of existing continual learning approaches. Furthermore, we showcase how LDC can be applied in combination with self-supervised CL methods, resulting in the first exemplar-free semi-supervised continual learning approach. We achieve state-of-the-art performance in both supervised and semi-supervised settings across multiple datasets. Code is available at \url{https://github.com/alviur/ldc}.