Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation
This addresses the problem of continual learning for AI systems with limited memory, though it is incremental as it builds on existing knowledge distillation techniques.
The paper tackles catastrophic forgetting in class-incremental learning by proposing a method that uses knowledge distillation and adaptive feature consolidation to balance robustness and flexibility, achieving significant accuracy improvements over existing methods on standard datasets.
We present a novel class incremental learning approach based on deep neural networks, which continually learns new tasks with limited memory for storing examples in the previous tasks. Our algorithm is based on knowledge distillation and provides a principled way to maintain the representations of old models while adjusting to new tasks effectively. The proposed method estimates the relationship between the representation changes and the resulting loss increases incurred by model updates. It minimizes the upper bound of the loss increases using the representations, which exploits the estimated importance of each feature map within a backbone model. Based on the importance, the model restricts updates of important features for robustness while allowing changes in less critical features for flexibility. This optimization strategy effectively alleviates the notorious catastrophic forgetting problem despite the limited accessibility of data in the previous tasks. The experimental results show significant accuracy improvement of the proposed algorithm over the existing methods on the standard datasets. Code is available.