Mass-Editing Memory in a Transformer
This addresses the need for efficient memory editing in language models, enabling broader updates beyond single associations, though it is incremental over prior editing methods.
The paper tackles the problem of updating large language models with many memories at once, developing MEMIT to scale up to thousands of associations for models like GPT-J and GPT-NeoX, exceeding prior work by orders of magnitude.
Recent work has shown exciting promise in updating large language models with new memories, so as to replace obsolete information or add specialized knowledge. However, this line of work is predominantly limited to updating single associations. We develop MEMIT, a method for directly updating a language model with many memories, demonstrating experimentally that it can scale up to thousands of associations for GPT-J (6B) and GPT-NeoX (20B), exceeding prior work by orders of magnitude. Our code and data are at https://memit.baulab.info.