Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
This highlights a critical privacy issue for users of AI assistants, though it is incremental as it builds on existing privacy concerns with a new benchmark.
The paper tackled the problem of privacy risks in large language models (LLMs) used interactively, showing that models like GPT-4 and ChatGPT reveal private information in 39% and 57% of cases, respectively, even with privacy prompts.
The interactive use of large language models (LLMs) in AI assistants (at work, home, etc.) introduces a new set of inference-time privacy risks: LLMs are fed different types of information from multiple sources in their inputs and are expected to reason about what to share in their outputs, for what purpose and with whom, within a given context. In this work, we draw attention to the highly critical yet overlooked notion of contextual privacy by proposing ConfAIde, a benchmark designed to identify critical weaknesses in the privacy reasoning capabilities of instruction-tuned LLMs. Our experiments show that even the most capable models such as GPT-4 and ChatGPT reveal private information in contexts that humans would not, 39% and 57% of the time, respectively. This leakage persists even when we employ privacy-inducing prompts or chain-of-thought reasoning. Our work underscores the immediate need to explore novel inference-time privacy-preserving approaches, based on reasoning and theory of mind.