Forgetting of task-specific knowledge in model merging-based continual learning
This addresses forgetting in continual learning for AI systems, but it is incremental as it builds on existing merging methods.
The paper investigates linear merging of models in continual learning, showing that merging preserves shared knowledge but degrades task-specific knowledge, and that merging incrementally trained models outperforms merging parallel-trained models.
This paper investigates the linear merging of models in the context of continual learning (CL). Using controlled visual cues in computer vision experiments, we demonstrate that merging largely preserves or enhances shared knowledge, while unshared task-specific knowledge rapidly degrades. We further find that merging models from an incremental training process consistently outperforms merging models trained in parallel.