Less-forgetting Learning in Deep Neural Networks
This addresses the problem of forgetting in neural networks for AI practitioners, but it appears incremental as it builds on existing research without a major paradigm shift.
The paper tackles catastrophic forgetting in deep neural networks when learning from new data, proposing a method that avoids using source domain information and reduces forgetting, also addressing forgetting between mini-batches to improve generalization and recognition rates.
A catastrophic forgetting problem makes deep neural networks forget the previously learned information, when learning data collected in new environments, such as by different sensors or in different light conditions. This paper presents a new method for alleviating the catastrophic forgetting problem. Unlike previous research, our method does not use any information from the source domain. Surprisingly, our method is very effective to forget less of the information in the source domain, and we show the effectiveness of our method using several experiments. Furthermore, we observed that the forgetting problem occurs between mini-batches when performing general training processes using stochastic gradient descent methods, and this problem is one of the factors that degrades generalization performance of the network. We also try to solve this problem using the proposed method. Finally, we show our less-forgetting learning method is also helpful to improve the performance of deep neural networks in terms of recognition rates.