Self-Supervised Training Enhances Online Continual Learning
This work addresses the challenge of incremental learning in non-stationary data streams for AI systems, showing an incremental improvement in performance for online continual learning tasks.
The paper tackled the problem of catastrophic forgetting in online continual learning by testing self-supervised pre-training methods like MoCo-V2, Barlow Twins, and SwAV on ImageNet, finding they outperform supervised pre-training, especially with fewer samples, and achieving a 14.95% relative increase in top-1 accuracy over prior state-of-the-art.
In continual learning, a system must incrementally learn from a non-stationary data stream without catastrophic forgetting. Recently, multiple methods have been devised for incrementally learning classes on large-scale image classification tasks, such as ImageNet. State-of-the-art continual learning methods use an initial supervised pre-training phase, in which the first 10% - 50% of the classes in a dataset are used to learn representations in an offline manner before continual learning of new classes begins. We hypothesize that self-supervised pre-training could yield features that generalize better than supervised learning, especially when the number of samples used for pre-training is small. We test this hypothesis using the self-supervised MoCo-V2, Barlow Twins, and SwAV algorithms. On ImageNet, we find that these methods outperform supervised pre-training considerably for online continual learning, and the gains are larger when fewer samples are available. Our findings are consistent across three online continual learning algorithms. Our best system achieves a 14.95% relative increase in top-1 accuracy on class incremental ImageNet over the prior state of the art for online continual learning.