EAMET: Robust Massive Model Editing via Embedding Alignment Optimization
This addresses the challenge of efficiently updating knowledge in LLMs for practical applications, though it appears incremental as it builds on existing model editing techniques.
The paper tackles the problem of robustly editing knowledge in large language models at scale, where existing methods degrade in massive editing scenarios, and proposes EAMET, which achieves about 90% editing efficacy when editing 10k facts.
Model editing techniques are essential for efficiently updating knowledge in large language models (LLMs). However, the effectiveness of existing approaches degrades in massive editing scenarios, particularly when evaluated with practical metrics. Their robustness is also limited in context-rich settings or when editing multiple facts of the same subject simultaneously. We attribute these failures to the embedding misalignment among knowledge items, which undermines editing reliability at scale. To address this, we propose EAMET (Embedding Alignment Model Editing in Transformers), which addresses this issue by aligning the space of key and residual embeddings. Extensive experiments across six LLMs and three datasets demonstrate that EAMET consistently outperforms existing methods, achieving about 90\% editing efficacy when editing 10k facts. Codes and datasets are publicly available at https://ybdai7.github.io/eamet-page/.