AKEW: Assessing Knowledge Editing in the Wild
This addresses the need for more practical evaluations in knowledge editing for AI researchers, though it is incremental as it builds on existing benchmarks.
The paper tackles the problem that current knowledge editing evaluations for language models are unrealistic by proposing AKEW, a new benchmark that includes structured facts, unstructured texts, and extracted triplets, and shows a considerable gap between state-of-the-art methods and practical scenarios.
Knowledge editing injects knowledge updates into language models to keep them correct and up-to-date. However, its current evaluations deviate significantly from practice: their knowledge updates solely consist of structured facts derived from meticulously crafted datasets, instead of practical sources -- unstructured texts like news articles, and they often overlook practical real-world knowledge updates. To address these issues, in this paper we propose AKEW (Assessing Knowledge Editing in the Wild), a new practical benchmark for knowledge editing. AKEW fully covers three editing settings of knowledge updates: structured facts, unstructured texts as facts, and extracted triplets. It further introduces new datasets featuring both counterfactual and real-world knowledge updates. Through extensive experiments, we demonstrate the considerable gap between state-of-the-art knowledge-editing methods and practical scenarios. Our analyses further highlight key insights to motivate future research for practical knowledge editing.