Long-form evaluation of model editing
This work addresses a gap in model editing evaluation for researchers and practitioners by providing a more comprehensive assessment in long-form settings, though it is incremental as it extends existing metrics.
The authors tackled the problem of evaluating model editing methods in long-form natural language generation, introducing LEME as a novel evaluation protocol that reveals new dimensions and benchmarks existing techniques, finding that methods like ROME and MEMIT perform well in limited edits but suffer from factual drift.
Evaluations of model editing currently only use the `next few token' completions after a prompt. As a result, the impact of these methods on longer natural language generation is largely unknown. We introduce long-form evaluation of model editing (LEME) a novel evaluation protocol that measures the efficacy and impact of model editing in long-form generative settings. Our protocol consists of a machine-rated survey and a classifier which correlates well with human ratings. Importantly, we find that our protocol has very little relationship with previous short-form metrics (despite being designed to extend efficacy, generalization, locality, and portability into a long-form setting), indicating that our method introduces a novel set of dimensions for understanding model editing methods. Using this protocol, we benchmark a number of model editing techniques and present several findings including that, while some methods (ROME and MEMIT) perform well in making consistent edits within a limited scope, they suffer much more from factual drift than other methods. Finally, we present a qualitative analysis that illustrates common failure modes in long-form generative settings including internal consistency, lexical cohesion, and locality issues.