Online Continual Knowledge Learning for Language Models
This addresses the issue of knowledge obsolescence in language models for applications like question-answering and fact-checking, but it is incremental as it builds on continual learning concepts.
The paper tackles the problem of keeping large language models up-to-date with changing world knowledge by introducing Online Continual Knowledge Learning (OCKL), and finds that existing continual learning methods are insufficient for this task, identifying key factors in the trade-off between acquiring new knowledge and retaining old knowledge.
Large Language Models (LLMs) serve as repositories of extensive world knowledge, enabling them to perform tasks such as question-answering and fact-checking. However, this knowledge can become obsolete as global contexts change. In this paper, we introduce a novel problem in the realm of continual learning: Online Continual Knowledge Learning (OCKL). This problem formulation aims to manage the dynamic nature of world knowledge in LMs under real-time constraints. We propose a new benchmark and evaluation metric designed to measure both the rate of new knowledge acquisition and the retention of previously learned knowledge. Our empirical evaluation, conducted using a variety of state-of-the-art methods, establishes robust base-lines for OCKL. Our results reveal that existing continual learning approaches are unfortunately insufficient for tackling the unique challenges posed by OCKL. We identify key factors that influence the trade-off between knowledge acquisition and retention, thereby advancing our understanding of how to train LMs in a continually evolving environment.