Continual Learning Beyond Experience Rehearsal and Full Model Surrogates
This addresses the scalability and efficiency limitations of continual learning methods for real-world applications, representing a novel approach rather than an incremental improvement.
The paper tackles the problem of catastrophic forgetting in continual learning by proposing SPARC, a method that eliminates the need for experience rehearsal and full-model surrogates, achieving superior performance on Seq-TinyImageNet and matching rehearsal-based methods while using only 6% of the parameters required by full-model surrogates.
Continual learning (CL) has remained a significant challenge for deep neural networks as learning new tasks erases previously acquired knowledge, either partially or completely. Existing solutions often rely on experience rehearsal or full model surrogates to mitigate CF. While effective, these approaches introduce substantial memory and computational overhead, limiting their scalability and applicability in real-world scenarios. To address this, we propose SPARC, a scalable CL approach that eliminates the need for experience rehearsal and full-model surrogates. By effectively combining task-specific working memories and task-agnostic semantic memory for cross-task knowledge consolidation, SPARC results in a remarkable parameter efficiency, using only 6% of the parameters required by full-model surrogates. Despite its lightweight design, SPARC achieves superior performance on Seq-TinyImageNet and matches rehearsal-based methods on various CL benchmarks. Additionally, weight re-normalization in the classification layer mitigates task-specific biases, establishing SPARC as a practical and scalable solution for CL under stringent efficiency constraints.