WikiBigEdit: Understanding the Limits of Lifelong Knowledge Editing in LLMs
This work addresses the challenge of efficiently updating knowledge in deployed LLMs for users relying on accurate information, though it is incremental as it builds on existing editing methods.
The paper tackled the problem of keeping large language models factually up-to-date by studying lifelong knowledge editing at a practical scale, introducing WikiBigEdit, a benchmark with over 500K question-answer pairs, and evaluating existing techniques against real-world edits.
Keeping large language models factually up-to-date is crucial for deployment, yet costly retraining remains a challenge. Knowledge editing offers a promising alternative, but methods are only tested on small-scale or synthetic edit benchmarks. In this work, we aim to bridge research into lifelong knowledge editing to real-world edits at a practically relevant scale. We first introduce WikiBigEdit; a large-scale benchmark of real-world Wikidata edits, built to automatically extend lifelong for future-proof benchmarking. In its first instance, it includes over 500K question-answer pairs for knowledge editing alongside a comprehensive evaluation pipeline. Finally, we use WikiBigEdit to study existing knowledge editing techniques' ability to incorporate large volumes of real-world facts and contrast their capabilities to generic modification techniques such as retrieval augmentation and continual finetuning to acquire a complete picture of the practical extent of current lifelong knowledge editing.