Mnemonics Training: Multi-Class Incremental Learning without Forgetting
This addresses the challenge of learning new concepts without forgetting old ones in incremental learning, which is crucial for applications like lifelong learning systems, though it is an incremental improvement over existing exemplar-based methods.
The paper tackles the problem of catastrophic forgetting in Multi-Class Incremental Learning (MCIL) by proposing an automatic framework called mnemonics that parameterizes and optimizes exemplars through bilevel optimization, achieving state-of-the-art results on benchmarks like CIFAR-100, ImageNet-Subset, and ImageNet with significant performance gains.
Multi-Class Incremental Learning (MCIL) aims to learn new concepts by incrementally updating a model trained on previous concepts. However, there is an inherent trade-off to effectively learning new concepts without catastrophic forgetting of previous ones. To alleviate this issue, it has been proposed to keep around a few examples of the previous concepts but the effectiveness of this approach heavily depends on the representativeness of these examples. This paper proposes a novel and automatic framework we call mnemonics, where we parameterize exemplars and make them optimizable in an end-to-end manner. We train the framework through bilevel optimizations, i.e., model-level and exemplar-level. We conduct extensive experiments on three MCIL benchmarks, CIFAR-100, ImageNet-Subset and ImageNet, and show that using mnemonics exemplars can surpass the state-of-the-art by a large margin. Interestingly and quite intriguingly, the mnemonics exemplars tend to be on the boundaries between different classes.