Can We Edit Factual Knowledge by In-Context Learning?
This addresses the challenge of updating knowledge in black-box LLMs efficiently, though it is incremental as it builds on in-context learning for a specific task.
The paper tackles the problem of editing factual knowledge in large language models (LLMs) without gradient-based fine-tuning, proposing in-context knowledge editing (IKE) which achieves a competitive success rate on GPT-J (6B) with fewer side effects like over-editing and knowledge forgetting, and scales to models like OPT-175B.
Previous studies have shown that large language models (LLMs) like GPTs store massive factual knowledge in their parameters. However, the stored knowledge could be false or out-dated. Traditional knowledge editing methods refine LLMs via fine-tuning on texts containing specific knowledge. However, with the increasing scales of LLMs, these gradient-based approaches bring large computation costs. The trend of model-as-a-service also makes it impossible to modify knowledge in black-box LMs. Inspired by in-context learning (ICL), a new paradigm based on demonstration contexts without parameter updating, we explore whether ICL can edit factual knowledge. To answer this question, we give a comprehensive empirical study of ICL strategies. Experiments show that in-context knowledge editing (IKE), without any gradient and parameter updating, achieves a competitive success rate compared to gradient-based methods on GPT-J (6B) but with much fewer side effects, including less over-editing on similar but unrelated facts and less knowledge forgetting on previously stored knowledge. We also apply the method to larger LMs with tens or hundreds of parameters like OPT-175B, which shows the scalability of our method. The code is available at https://github.com/Zce1112zslx/IKE.