DeltaEdit: Enhancing Sequential Editing in Large Language Models by Controlling Superimposed Noise
This addresses the issue of maintaining up-to-date and accurate knowledge in large language models during long-term editing, which is incremental as it builds on existing sequential editing methods.
The paper tackles the problem of declining success rates in sequential knowledge editing for large language models by identifying the accumulation of superimposed noise as the cause. They propose DeltaEdit, which uses dynamic orthogonal constraints to reduce interference between edits, resulting in significantly improved edit success rates and better retention of generalization capabilities.
Sequential knowledge editing techniques aim to continuously update the knowledge in large language models at a low cost, preventing the models from generating outdated or incorrect information. However, existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing. Through theoretical analysis and experiments, we identify that as the number of edits increases, the model's output increasingly deviates from the desired target, leading to a drop in editing success rates. We refer to this issue as the accumulation of superimposed noise problem. To address this, we identify the factors contributing to this deviation and propose DeltaEdit, a novel method that optimizes update parameters through a dynamic orthogonal constraints strategy, effectively reducing interference between edits to mitigate deviation. Experimental results demonstrate that DeltaEdit significantly outperforms existing methods in edit success rates and the retention of generalization capabilities, ensuring stable and reliable model performance even under extensive sequential editing.