Catastrophic Forgetting in Deep Learning: A Comprehensive Taxonomy
This is an incremental survey paper that addresses the challenge of incremental learning for AI researchers and practitioners, aiming to improve model adaptability without forgetting.
The paper tackles the problem of catastrophic forgetting in deep learning, where models lose accuracy on old tasks when learning new ones, by surveying recent solutions and proposing a taxonomy to organize them, but does not present new experimental results or concrete numbers.
Deep Learning models have achieved remarkable performance in tasks such as image classification or generation, often surpassing human accuracy. However, they can struggle to learn new tasks and update their knowledge without access to previous data, leading to a significant loss of accuracy known as Catastrophic Forgetting (CF). This phenomenon was first observed by McCloskey and Cohen in 1989 and remains an active research topic. Incremental learning without forgetting is widely recognized as a crucial aspect in building better AI systems, as it allows models to adapt to new tasks without losing the ability to perform previously learned ones. This article surveys recent studies that tackle CF in modern Deep Learning models that use gradient descent as their learning algorithm. Although several solutions have been proposed, a definitive solution or consensus on assessing CF is yet to be established. The article provides a comprehensive review of recent solutions, proposes a taxonomy to organize them, and identifies research gaps in this area.