Overcoming Multi-Model Forgetting
This addresses a practical issue for researchers and practitioners in deep learning who train multiple models sequentially, offering an incremental improvement to mitigate forgetting in shared-parameter settings.
The paper tackles the problem of multi-model forgetting, where sequentially training multiple deep networks with shared parameters degrades the performance of earlier models, and introduces a weight plasticity loss that regularizes learning based on parameter importance, showing effectiveness in sequential training and neural architecture search with improved results in NLP and computer vision tasks.
We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, we introduce a statistically-justified weight plasticity loss that regularizes the learning of a model's shared parameters according to their importance for the previous models, and demonstrate its effectiveness when training two models sequentially and for neural architecture search. Adding weight plasticity in neural architecture search preserves the best models to the end of the search and yields improved results in both natural language processing and computer vision tasks.