Fixed Design Analysis of Regularization-Based Continual Learning
This work addresses catastrophic forgetting in continual learning for machine learning practitioners, but it is incremental as it focuses on a specific theoretical setting.
The paper tackles the problem of catastrophic forgetting in continual learning by analyzing an ℓ₂-regularized algorithm for two linear regression tasks, providing tight bounds on average risk that reveal a trade-off between forgetting and intransigence, with results showing that well-tuned regularization can partially mitigate forgetting for dissimilar tasks.
We consider a continual learning (CL) problem with two linear regression tasks in the fixed design setting, where the feature vectors are assumed fixed and the labels are assumed to be random variables. We consider an $\ell_2$-regularized CL algorithm, which computes an Ordinary Least Squares parameter to fit the first dataset, then computes another parameter that fits the second dataset under an $\ell_2$-regularization penalizing its deviation from the first parameter, and outputs the second parameter. For this algorithm, we provide tight bounds on the average risk over the two tasks. Our risk bounds reveal a provable trade-off between forgetting and intransigence of the $\ell_2$-regularized CL algorithm: with a large regularization parameter, the algorithm output forgets less information about the first task but is intransigent to extract new information from the second task; and vice versa. Our results suggest that catastrophic forgetting could happen for CL with dissimilar tasks (under a precise similarity measurement) and that a well-tuned $\ell_2$-regularization can partially mitigate this issue by introducing intransigence.