Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuning
This work addresses the incremental challenge of updating language models with new information for AI researchers and practitioners, offering a dataset and method to improve knowledge acquisition.
The authors tackled the problem of teaching new world knowledge facts to pre-trained language models, which is challenging due to slow learning rates and catastrophic forgetting. They introduced the Novel-WD dataset from recent Wikidata updates and demonstrated that prefix-tuning can reliably encode a single fact per prefix, with capacity increasing with prefix length and model size.
Teaching new information to pre-trained large language models (PLM) is a crucial but challenging task. Model adaptation techniques, such as fine-tuning and parameter-efficient training have been shown to store new facts at a slow rate; continual learning is an option but is costly and prone to catastrophic forgetting. This work studies and quantifies how PLM may learn and remember new world knowledge facts that do not occur in their pre-training corpus, which only contains world knowledge up to a certain date. To that purpose, we first propose Novel-WD, a new dataset consisting of sentences containing novel facts extracted from recent Wikidata updates, along with two evaluation tasks in the form of causal language modeling and multiple choice questions (MCQ). We make this dataset freely available to the community, and release a procedure to later build new versions of similar datasets with up-to-date information. We also explore the use of prefix-tuning for novel information learning, and analyze how much information can be stored within a given prefix. We show that a single fact can reliably be encoded within a single prefix, and that the prefix capacity increases with its length and with the base model size.