AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training
This addresses the problem of catastrophic forgetting in deep neural networks for continual learning, offering a simple and effective solution that reduces hyperparameter tuning and computational costs, though it appears incremental as it builds on existing flat minima approaches.
The paper tackles catastrophic forgetting in continual learning by proposing AlterSGD, an optimization method that alternates gradient descent and ascent to find flat minima, resulting in significant mitigation of forgetting and outperforming state-of-the-art methods by a large margin on semantic segmentation benchmarks.
Deep neural networks suffer from catastrophic forgetting when learning multiple knowledge sequentially, and a growing number of approaches have been proposed to mitigate this problem. Some of these methods achieved considerable performance by associating the flat local minima with forgetting mitigation in continual learning. However, they inevitably need (1) tedious hyperparameters tuning, and (2) additional computational cost. To alleviate these problems, in this paper, we propose a simple yet effective optimization method, called AlterSGD, to search for a flat minima in the loss landscape. In AlterSGD, we conduct gradient descent and ascent alternatively when the network tends to converge at each session of learning new knowledge. Moreover, we theoretically prove that such a strategy can encourage the optimization to converge to a flat minima. We verify AlterSGD on continual learning benchmark for semantic segmentation and the empirical results show that we can significantly mitigate the forgetting and outperform the state-of-the-art methods with a large margin under challenging continual learning protocols.