InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing
This work addresses efficiency and scalability issues in model editing for LLM users, though it is incremental as it builds on existing in-context learning methods.
The paper tackles the problem of limited context window in in-context learning for model editing in large language models, which degrades performance with many edits, and proposes InComeS, a framework that integrates compression and selection mechanisms to enhance efficiency and effectiveness, as demonstrated by experiments on diverse benchmarks.
Although existing model editing methods perform well in recalling exact edit facts, they often struggle in complex scenarios that require deeper semantic understanding rather than mere knowledge regurgitation. Leveraging the strong contextual reasoning abilities of large language models (LLMs), in-context learning (ICL) becomes a promising editing method by comprehending edit information through context encoding. However, this method is constrained by the limited context window of LLMs, leading to degraded performance and efficiency as the number of edits increases. To overcome this limitation, we propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts through explicit compression and selection mechanisms. Specifically, InComeS compresses each editing context into the key-value (KV) cache of a special gist token, enabling efficient handling of multiple edits without being restricted by the model's context window. Furthermore, specialized cross-attention modules are added to dynamically select the most relevant information from the gist pools, enabling adaptive and effective utilization of edit information. We conduct experiments on diverse model editing benchmarks with various editing formats, and the results demonstrate the effectiveness and efficiency of our method.