Continual Pre-Training Mitigates Forgetting in Language and Vision
This addresses the issue of forgetting in machine learning models for researchers and practitioners in continual learning, though it is incremental as it builds on existing pre-training and continual learning concepts.
The paper tackles the problem of catastrophic forgetting in continual learning by formalizing and investigating continual pre-training on streams of data before fine-tuning, showing that it mitigates forgetting and that self-supervised pre-training is more effective than supervised methods in retaining knowledge.
Pre-trained models are nowadays a fundamental component of machine learning research. In continual learning, they are commonly used to initialize the model before training on the stream of non-stationary data. However, pre-training is rarely applied during continual learning. We formalize and investigate the characteristics of the continual pre-training scenario in both language and vision environments, where a model is continually pre-trained on a stream of incoming data and only later fine-tuned to different downstream tasks. We show that continually pre-trained models are robust against catastrophic forgetting and we provide strong empirical evidence supporting the fact that self-supervised pre-training is more effective in retaining previous knowledge than supervised protocols. Code is provided at https://github.com/AndreaCossu/continual-pretraining-nlp-vision .