On the Privacy Risk of In-context Learning
This addresses a critical privacy problem for companies and users deploying LLMs with sensitive data, highlighting a novel vulnerability in a widely used technique.
The paper tackles the privacy risk of using private data in prompts for in-context learning with large language models, showing that this approach enables highly effective membership inference attacks and poses greater risks than fine-tuning at similar utility levels.
Large language models (LLMs) are excellent few-shot learners. They can perform a wide variety of tasks purely based on natural language prompts provided to them. These prompts contain data of a specific downstream task -- often the private dataset of a party, e.g., a company that wants to leverage the LLM for their purposes. We show that deploying prompted models presents a significant privacy risk for the data used within the prompt by instantiating a highly effective membership inference attack. We also observe that the privacy risk of prompted models exceeds fine-tuned models at the same utility levels. After identifying the model's sensitivity to their prompts -- in the form of a significantly higher prediction confidence on the prompted data -- as a cause for the increased risk, we propose ensembling as a mitigation strategy. By aggregating over multiple different versions of a prompted model, membership inference risk can be decreased.