MEMIT-Merge: Addressing MEMIT's Key-Value Conflicts in Same-Subject Batch Editing for LLMs
This work addresses a significant problem for developers and users of large language models, particularly those relying on MEMIT for batch editing, by providing a more robust and efficient solution for same-subject batch editing scenarios, which is an incremental improvement over existing methods.
The authors tackled the problem of key-value conflicts in MEMIT, a batch editing algorithm for large language models, and achieved a success rate exceeding 90% with their proposed MEMIT-Merge approach, outperforming the original MEMIT which drops to around 50% at larger batch sizes. This improvement showcases remarkable robustness to subject entity collisions.
As large language models continue to scale up, knowledge editing techniques that modify models' internal knowledge without full retraining have gained significant attention. MEMIT, a prominent batch editing algorithm, stands out for its capability to perform mass knowledge modifications. However, we uncover that MEMIT's editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject. Our analysis reveals this stems from MEMIT's key value modeling framework: identical keys (derived from the shared subject) are forced to represent different values (corresponding to different knowledge), resulting in update conflicts during editing. Addressing this issue, we propose MEMIT-Merge, an enhanced approach that merges value computation processes for facts sharing the same subject, effectively resolving the performance degradation in samesubject batch editing scenarios. Experimental results demonstrate that when MEMIT's edit success rate drops to around 50% at larger batch sizes, MEMIT-Merge maintains a success rate exceeding 90%, showcasing remarkable robustness to subject entity collisions. The code is available at https://github.com/NUSTM/ MEMIT-Merge.