Adapt & Align: Continual Learning with Generative Models Latent Space Alignment
This addresses the issue of performance degradation in neural networks when trained on new data distributions without access to old examples, which is a key challenge in continual learning for AI systems.
The paper tackles the problem of catastrophic forgetting in neural networks during continual learning by introducing Adapt & Align, a method that aligns latent representations in generative models, resulting in improved performance on downstream tasks like classification.
In this work, we introduce Adapt & Align, a method for continual learning of neural networks by aligning latent representations in generative models. Neural Networks suffer from abrupt loss in performance when retrained with additional training data from different distributions. At the same time, training with additional data without access to the previous examples rarely improves the model's performance. In this work, we propose a new method that mitigates those problems by employing generative models and splitting the process of their update into two parts. In the first one, we train a local generative model using only data from a new task. In the second phase, we consolidate latent representations from the local model with a global one that encodes knowledge of all past experiences. We introduce our approach with Variational Auteoncoders and Generative Adversarial Networks. Moreover, we show how we can use those generative models as a general method for continual knowledge consolidation that can be used in downstream tasks such as classification.