On the Implicit Adversariality of Catastrophic Forgetting in Deep Continual Learning
This addresses the problem of catastrophic forgetting for continual learning systems, offering a novel theoretical insight and method, though it is incremental as it builds on existing gradient projection approaches.
The paper tackles catastrophic forgetting in deep continual learning by revealing that new-task training acts as an implicit adversarial attack on old-task knowledge, and proposes backGP to address this, reducing forgetting by 10.8% and improving accuracy by 12.7% on average over gradient projection methods.
Continual learning seeks the human-like ability to accumulate new skills in machine intelligence. Its central challenge is catastrophic forgetting, whose underlying cause has not been fully understood for deep networks. In this paper, we demystify catastrophic forgetting by revealing that the new-task training is implicitly an adversarial attack against the old-task knowledge. Specifically, the new-task gradients automatically and accurately align with the sharp directions of the old-task loss landscape, rapidly increasing the old-task loss. This adversarial alignment is intriguingly counter-intuitive because the sharp directions are too sparsely distributed to align with by chance. To understand it, we theoretically show that it arises from training's low-rank bias, which, through forward and backward propagation, confines the two directions into the same low-dimensional subspace, facilitating alignment. Gradient projection (GP) methods, a representative family of forgetting-mitigating methods, reduce adversarial alignment caused by forward propagation, but cannot address the alignment due to backward propagation. We propose backGP to address it, which reduces forgetting by 10.8% and improves accuracy by 12.7% on average over GP methods.