Toward Ultra-Long-Horizon Sequential Model Editing
This addresses a critical bottleneck for practitioners needing to maintain updated and accurate LLMs over many edits, though it is an incremental improvement to existing methods.
The paper tackled the problem of model collapse in sequential editing of large language models by identifying explosive weight norm growth as the cause and proposing a norm-constrained strategy, which delayed collapse by over 4 times and improved editing performance by 72.2% on average.
Model editing has emerged as a practical approach for mitigating factual errors and outdated knowledge in large language models (LLMs). Among existing methods, the Locate-and-Edit (L&E) paradigm is the dominant framework: it locates MLP parameters implicated in expressing a target fact, and then performs a localized update to rewrite that fact. However, long sequences of edits often trigger abrupt model collapse in L&E beyond a critical point. We empirically identify a strong correlation between collapse and explosive growth of edited MLP weight norms, and formally prove that commonly used L&E update rules can induce exponential norm growth across sequential edits in the absence of explicit norm control. To address this issue, we propose Norm-Anchor Scaling NAS, a plug-and-play norm-constrained strategy. Across extensive experiments, NAS delays the collapse point of representative L&E algorithms by more than 4 times and yields a 72.2% average relative gain in editing performance, requiring only a single additional line of code and incurring negligible computational overhead.