SPeCiaL: Self-Supervised Pretraining for Continual Learning
This addresses the challenge of forgetting in continual learning for AI systems that need to learn sequentially, though it appears incremental as it builds on existing self-supervised and meta-learning techniques.
The paper tackles the problem of catastrophic forgetting in continual learning by developing SPeCiaL, a self-supervised pretraining method that uses a meta-learning objective to train representations for quick knowledge retention. The result shows it matches or outperforms supervised pretraining approaches in Continual Few-Shot Learning.
This paper presents SPeCiaL: a method for unsupervised pretraining of representations tailored for continual learning. Our approach devises a meta-learning objective that differentiates through a sequential learning process. Specifically, we train a linear model over the representations to match different augmented views of the same image together, each view presented sequentially. The linear model is then evaluated on both its ability to classify images it just saw, and also on images from previous iterations. This gives rise to representations that favor quick knowledge retention with minimal forgetting. We evaluate SPeCiaL in the Continual Few-Shot Learning setting, and show that it can match or outperform other supervised pretraining approaches.