Overcoming Catastrophic Forgetting with Unlabeled Data in the Wild
This addresses the problem of performance degradation in AI systems when learning new tasks, offering a practical solution for incremental learning scenarios, though it is incremental in nature.
The paper tackles catastrophic forgetting in lifelong learning with deep neural networks by leveraging unlabeled data from the wild, achieving up to 15.8% higher accuracy and 46.5% less forgetting compared to state-of-the-art methods on datasets like CIFAR and ImageNet.
Lifelong learning with deep neural networks is well-known to suffer from catastrophic forgetting: the performance on previous tasks drastically degrades when learning a new task. To alleviate this effect, we propose to leverage a large stream of unlabeled data easily obtainable in the wild. In particular, we design a novel class-incremental learning scheme with (a) a new distillation loss, termed global distillation, (b) a learning strategy to avoid overfitting to the most recent task, and (c) a confidence-based sampling method to effectively leverage unlabeled external data. Our experimental results on various datasets, including CIFAR and ImageNet, demonstrate the superiority of the proposed methods over prior methods, particularly when a stream of unlabeled data is accessible: our method shows up to 15.8% higher accuracy and 46.5% less forgetting compared to the state-of-the-art method. The code is available at https://github.com/kibok90/iccv2019-inc.