RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World
This work addresses the challenge of continuous knowledge drift for LLMs in real-world applications, providing a foundation for better adaptation methods, though it is incremental in proposing a baseline solution.
The paper tackles the problem of large language models (LLMs) struggling to adapt to continuously evolving knowledge, which leads to outdated predictions and temporal inconsistencies. It introduces a new benchmark for evaluating adaptation methods and finds that most existing approaches fail, while proposing a time-aware retrieval baseline, Chronos, that improves temporal consistency without training.
Large language models (LLMs) acquire most of their knowledge during pretraining, which ties them to a fixed snapshot of the world and makes adaptation to continuously evolving knowledge challenging. As facts, entities, and events change over time, models may experience continuous knowledge drift, resulting not only in outdated predictions but also in temporally inconsistent reasoning. Although existing approaches, such as continual finetuning, knowledge editing, and retrieval-augmented generation (RAG), aim to update or supplement model knowledge, they are rarely evaluated in settings that reflect chronological, evolving, and real-world knowledge evolution. In this work, we introduce a new benchmark of real-world dynamic events, constructed from time-stamped evidence that captures how knowledge evolves over time, which enables systematic evaluation of model adaptation under continuous knowledge drift. The benchmark reveals that most existing methods, including vanilla RAG and several learning-based approaches, struggle under this setting, exposing critical limitations such as catastrophic forgetting and temporal inconsistency. To mitigate these limitations, we propose a time-aware retrieval baseline, Chronos, which progressively organizes retrieved evidence into an Event Evolution Graph to enable more temporally consistent understanding in LLMs without additional training. Overall, this work provides a foundation for analyzing and advancing LLM adaptation to continuous knowledge drift in realistic settings.