Convergence of Continual Learning in Homogeneous Deep Networks
This work provides a theoretical foundation for understanding convergence in continual learning, addressing a key gap for deep neural networks.
The paper characterizes continual learning in homogeneous deep networks as sequential projections onto task margin sets, proving that global convergence generally fails but local linear convergence is guaranteed under random and cyclic task sequences.
We characterize weakly regularized continual classification in homogeneous models as sequential projections onto task margin sets. This result generalizes prior analyses restricted to either stationary (single-task) deep models or continual linear models. We show that global convergence generally fails, even for simple models linear in data but nonlinear in parameters. Nevertheless, by leveraging results from nonconvex projection theory, we identify regularity properties of homogeneous deep networks that guarantee local linear convergence under random and cyclic task sequences. Finally, we extend our analysis to continual regression, unifying the framework for homogeneous models.