LOKI: Memory-Free Null-Space Constrained Lifelong Knowledge Editing
Addresses catastrophic forgetting in lifelong knowledge editing for language models, offering a more flexible and memory-free solution.
Lifelong knowledge editing updates language models sequentially without forgetting past knowledge. LOKI uses dynamic layer selection and null-space projection to achieve up to 14% improvement in average accuracy over existing methods.
Lifelong knowledge editing aims to efficiently and sequentially update language models over time, as new knowledge becomes available or when the model makes mistakes, while preserving acceptable performance on past knowledge. One unresolved challenge is that existing methods modify a fixed set of layers for all new knowledge samples, reducing flexibility and increasing catastrophic forgetting. Another is requiring access to previous knowledge and extensive pre-processing to obtain data statistics. To address these challenges, we introduce LOKI, a novel approach that uses dynamic layer selection based on the Hilbert-Schmidt Independence Criterion and projects gradient updates onto the null-space of the model weights, bypassing the requirement for previous knowledge access. We show that LOKI achieves superior performance to existing approaches across a wide variety of experiments, achieving up to a 14\% improvement in average accuracy.