A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initialization
This addresses computational inefficiency in continual learning systems for real-world data streams, though it is incremental as it builds on existing Neural Collapse insights.
The paper tackles the problem of high initial training loss and instability in continual learning when adding new categories by proposing a data-driven weight initialization strategy based on Neural Collapse, which mitigates loss spikes and accelerates adaptation to new tasks, demonstrating faster adaptation and improved performance in large-scale settings.
To adapt to real-world data streams, continual learning (CL) systems must rapidly learn new concepts while preserving and utilizing prior knowledge. When it comes to adding new information to continually-trained deep neural networks (DNNs), classifier weights for newly encountered categories are typically initialized randomly, leading to high initial training loss (spikes) and instability. Consequently, achieving optimal convergence and accuracy requires prolonged training, increasing computational costs. Inspired by Neural Collapse (NC), we propose a weight initialization strategy to improve learning efficiency in CL. In DNNs trained with mean-squared-error, NC gives rise to a Least-Square (LS) classifier in the last layer, whose weights can be analytically derived from learned features. We leverage this LS formulation to initialize classifier weights in a data-driven manner, aligning them with the feature distribution rather than using random initialization. Our method mitigates initial loss spikes and accelerates adaptation to new tasks. We evaluate our approach in large-scale CL settings, demonstrating faster adaptation and improved CL performance.