Heterogeneous Continual Learning
This addresses the challenge of adapting continual learning solutions to evolving architectures, which is incremental but relevant for machine learning practitioners dealing with dynamic model deployments.
The paper tackles the problem of continual learning with changing network architectures by proposing Heterogeneous Continual Learning (HCL), which uses a modified knowledge distillation approach and Quick Deep Inversion (QDI) to recover prior task features, resulting in significant accuracy improvements over state-of-the-art methods on various benchmarks.
We propose a novel framework and a solution to tackle the continual learning (CL) problem with changing network architectures. Most CL methods focus on adapting a single architecture to a new task/class by modifying its weights. However, with rapid progress in architecture design, the problem of adapting existing solutions to novel architectures becomes relevant. To address this limitation, we propose Heterogeneous Continual Learning (HCL), where a wide range of evolving network architectures emerge continually together with novel data/tasks. As a solution, we build on top of the distillation family of techniques and modify it to a new setting where a weaker model takes the role of a teacher; meanwhile, a new stronger architecture acts as a student. Furthermore, we consider a setup of limited access to previous data and propose Quick Deep Inversion (QDI) to recover prior task visual features to support knowledge transfer. QDI significantly reduces computational costs compared to previous solutions and improves overall performance. In summary, we propose a new setup for CL with a modified knowledge distillation paradigm and design a quick data inversion method to enhance distillation. Our evaluation of various benchmarks shows a significant improvement on accuracy in comparison to state-of-the-art methods over various networks architectures.