Momentum-based Weight Interpolation of Strong Zero-Shot Models for Continual Learning
This addresses the challenge of adapting zero-shot models for continual learning without sacrificing robustness, though it appears incremental as it builds on existing interpolation techniques.
The paper tackles the problem of large pre-trained zero-shot models losing generalizability and robustness when fine-tuned for continual learning tasks, and shows that momentum-based weight interpolation provides consistent improvements with over +4% on standard benchmarks and reduces the error to joint training limits by more than half.
Large pre-trained, zero-shot capable models have shown considerable success both for standard transfer and adaptation tasks, with particular robustness towards distribution shifts. In addition, subsequent fine-tuning can considerably improve performance on a selected downstream task. However, through naive fine-tuning, these zero-shot models lose their generalizability and robustness towards distribution shifts. This is a particular problem for tasks such as Continual Learning (CL), where continuous adaptation has to be performed as new task distributions are introduced sequentially. In this work, we showcase that where fine-tuning falls short to adapt such zero-shot capable models, simple momentum-based weight interpolation can provide consistent improvements for CL tasks in both memory-free and memory-based settings. In particular, we find improvements of over $+4\%$ on standard CL benchmarks, while reducing the error to the upper limit of jointly training on all tasks at once in parts by more than half, allowing the continual learner to inch closer to the joint training limits.