Learning without Memorizing
This addresses the memory inefficiency in incremental learning for edge devices, though it is an incremental improvement over existing methods.
The paper tackles the problem of incremental learning (IL) requiring storage of old class data, which is impractical for memory-limited edge devices, by proposing 'Learning without Memorizing (LwM)' with an Attention Distillation Loss that consistently outperforms state-of-the-art methods on iILSVRC-small and iCIFAR-100 datasets in overall accuracy.
Incremental learning (IL) is an important task aimed at increasing the capability of a trained model, in terms of the number of classes recognizable by the model. The key problem in this task is the requirement of storing data (e.g. images) associated with existing classes, while teaching the classifier to learn new classes. However, this is impractical as it increases the memory requirement at every incremental step, which makes it impossible to implement IL algorithms on edge devices with limited memory. Hence, we propose a novel approach, called `Learning without Memorizing (LwM)', to preserve the information about existing (base) classes, without storing any of their data, while making the classifier progressively learn the new classes. In LwM, we present an information preserving penalty: Attention Distillation Loss ($L_{AD}$), and demonstrate that penalizing the changes in classifiers' attention maps helps to retain information of the base classes, as new classes are added. We show that adding $L_{AD}$ to the distillation loss which is an existing information preserving loss consistently outperforms the state-of-the-art performance in the iILSVRC-small and iCIFAR-100 datasets in terms of the overall accuracy of base and incrementally learned classes.