Spectral Characterization and Mitigation of Sequential Knowledge Editing Collapse
This addresses the issue of model degradation during sequential editing for users of large language models, offering a novel mitigation approach.
The paper tackled the problem of catastrophic collapse in sequential knowledge editing for large language models by identifying that general abilities are linked to dominant singular directions in weight matrices, which are disrupted by repeated edits. They proposed REVIVE, a framework that preserves these directions, improving editing efficacy and maintaining general abilities in experiments with up to 20,000 edits.
Sequential knowledge editing in large language models often causes catastrophic collapse of the model's general abilities, especially for parameter-modifying methods. Existing approaches mitigate this issue through heuristic constraints on parameter updates, yet the mechanisms underlying such degradation remain insufficiently understood. In this work, we present a spectral analysis of sequential knowledge editing and show that a model's general abilities are closely associated with dominant singular directions of pretrained weight matrices. These directions are highly sensitive to perturbations and are progressively disrupted by repeated edits, closely tracking the collapse in both editing efficacy and general performance. Building on this insight, we propose REVIVE, a plug-and-play framework that stabilizes sequential editing by explicitly preserving the dominant singular subspace. REVIVE represents parameter updates in the spectral basis of the original weights and filters components that would interfere with the protected region. Extensive experiments across multiple models and benchmarks show that REVIVE consistently improves editing efficacy while substantially preserving general abilities under long-horizon sequential editing, including extreme settings with up to 20,000 edits.