ContextLeak: Auditing Leakage in Private In-Context Learning Methods
This addresses the need for reliable privacy auditing in machine learning applications handling sensitive data, though it is incremental as it builds on existing private ICL techniques.
The paper tackled the problem of auditing information leakage in private in-context learning methods for large language models, introducing ContextLeak as a framework to measure worst-case leakage and finding it correlates with theoretical privacy budgets while revealing poor privacy-utility trade-offs in existing methods.
In-Context Learning (ICL) has become a standard technique for adapting Large Language Models (LLMs) to specialized tasks by supplying task-specific exemplars within the prompt. However, when these exemplars contain sensitive information, reliable privacy-preserving mechanisms are essential to prevent unintended leakage through model outputs. Many privacy-preserving methods are proposed to protect the information leakage in the context, but there are less efforts on how to audit those methods. We introduce ContextLeak, the first framework to empirically measure the worst-case information leakage in ICL. ContextLeak uses canary insertion, embedding uniquely identifiable tokens in exemplars and crafting targeted queries to detect their presence. We apply ContextLeak across a range of private ICL techniques, both heuristic such as prompt-based defenses and those with theoretical guarantees such as Embedding Space Aggregation and Report Noisy Max. We find that ContextLeak tightly correlates with the theoretical privacy budget ($ε$) and reliably detects leakage. Our results further reveal that existing methods often strike poor privacy-utility trade-offs, either leaking sensitive information or severely degrading performance.