Continual Learning in Deep Networks: an Analysis of the Last Layer
This work addresses catastrophic forgetting in continual learning for AI practitioners, but it is incremental as it focuses on output layer parameterizations rather than introducing new algorithms.
The paper analyzes how different output layer parameterizations in deep neural networks affect learning and forgetting in continual learning, showing that selecting the best-performing layer type depends on data distribution drifts and data availability, with some cases achieving significantly better performance without specialized algorithms.
We study how different output layer parameterizations of a deep neural network affects learning and forgetting in continual learning settings. The following three effects can cause catastrophic forgetting in the output layer: (1) weights modifications, (2) interference, and (3) projection drift. In this paper, our goal is to provide more insights into how changing the output layer parameterization may address (1) and (2). Some potential solutions to those issues are proposed and evaluated here in several continual learning scenarios. We show that the best-performing type of output layer depends on the data distribution drifts and/or the amount of data available. In particular, in some cases where a standard linear layer would fail, changing parameterization is sufficient to achieve a significantly better performance, without introducing any continual-learning algorithm but instead by using standard SGD to train a model. Our analysis and results shed light on the dynamics of the output layer in continual learning scenarios and suggest a way of selecting the best type of output layer for a given scenario.