Continual Lifelong Learning in Natural Language Processing: A Survey
This survey addresses the problem of catastrophic forgetting in continual learning for NLP researchers and practitioners, providing a comprehensive overview of the current state and future directions.
This survey paper examines the challenges and current methods of continual learning (CL) in natural language processing (NLP), focusing on how deep learning architectures struggle to learn new tasks without forgetting prior knowledge. It reviews existing CL evaluation methods and datasets specific to NLP.
Continual learning (CL) aims to enable information systems to learn from a continuous data stream across time. However, it is difficult for existing deep learning architectures to learn a new task without largely forgetting previously acquired knowledge. Furthermore, CL is particularly challenging for language learning, as natural language is ambiguous: it is discrete, compositional, and its meaning is context-dependent. In this work, we look at the problem of CL through the lens of various NLP tasks. Our survey discusses major challenges in CL and current methods applied in neural network models. We also provide a critical review of the existing CL evaluation methods and datasets in NLP. Finally, we present our outlook on future research directions.