Analysis of Catastrophic Forgetting for Random Orthogonal Transformation Tasks in the Overparameterized Regime
This addresses catastrophic forgetting in neural networks for continual learning applications, but it is incremental as it builds on existing overparameterization and theoretical frameworks.
The paper tackles catastrophic forgetting in continual learning by analyzing the effect of overparameterization, showing experimentally that overparameterization improves generalization in permuted MNIST tasks with performance gains comparable to state-of-the-art algorithms, and theoretically explaining this through a two-task linear regression model where overparameterization reduces risk gain on the first task.
Overparameterization is known to permit strong generalization performance in neural networks. In this work, we provide an initial theoretical analysis of its effect on catastrophic forgetting in a continual learning setup. We show experimentally that in permuted MNIST image classification tasks, the generalization performance of multilayer perceptrons trained by vanilla stochastic gradient descent can be improved by overparameterization, and the extent of the performance increase achieved by overparameterization is comparable to that of state-of-the-art continual learning algorithms. We provide a theoretical explanation of this effect by studying a qualitatively similar two-task linear regression problem, where each task is related by a random orthogonal transformation. We show that when a model is trained on the two tasks in sequence without any additional regularization, the risk gain on the first task is small if the model is sufficiently overparameterized.