On the Theory of Continual Learning with Gradient Descent for Neural Networks
This work provides theoretical insights into continual learning mechanisms, but it is incremental as it builds on existing frameworks with a simplified model.
The paper tackled the problem of understanding forgetting in continual learning by analyzing gradient descent on a tractable neural network model, deriving bounds on forgetting rates in terms of key parameters like iterations and hidden-layer size, with numerical experiments validating the findings.
Continual learning, the ability of a model to adapt to an ongoing sequence of tasks without forgetting the earlier ones, is a central goal of artificial intelligence. To shed light on its underlying mechanisms, we analyze the limitations of continual learning in a tractable yet representative setting. In particular, we study one-hidden-layer quadratic neural networks trained by gradient descent on an XOR cluster dataset with Gaussian noise, where different tasks correspond to different clusters with orthogonal means. Our results obtain bounds on the rate of forgetting during train and test-time in terms of the number of iterations, the sample size, the number of tasks, and the hidden-layer size. Our results reveal interesting phenomena on the role of different problem parameters in the rate of forgetting. Numerical experiments across diverse setups confirm our results, demonstrating their validity beyond the analyzed settings.