Rethinking Momentum Knowledge Distillation in Online Continual Learning
This work addresses the problem of improving learning efficiency in online continual learning for AI systems, though it is incremental as it builds on existing replay-based strategies.
The paper tackled the challenge of applying Knowledge Distillation (KD) to Online Continual Learning (OCL), where data is seen only once, by introducing Momentum Knowledge Distillation (MKD) to enhance existing methods, resulting in over 10% accuracy improvement on ImageNet100.
Online Continual Learning (OCL) addresses the problem of training neural networks on a continuous data stream where multiple classification tasks emerge in sequence. In contrast to offline Continual Learning, data can be seen only once in OCL, which is a very severe constraint. In this context, replay-based strategies have achieved impressive results and most state-of-the-art approaches heavily depend on them. While Knowledge Distillation (KD) has been extensively used in offline Continual Learning, it remains under-exploited in OCL, despite its high potential. In this paper, we analyze the challenges in applying KD to OCL and give empirical justifications. We introduce a direct yet effective methodology for applying Momentum Knowledge Distillation (MKD) to many flagship OCL methods and demonstrate its capabilities to enhance existing approaches. In addition to improving existing state-of-the-art accuracy by more than $10\%$ points on ImageNet100, we shed light on MKD internal mechanics and impacts during training in OCL. We argue that similar to replay, MKD should be considered a central component of OCL. The code is available at \url{https://github.com/Nicolas1203/mkd_ocl}.