Teach LLMs to Phish: Stealing Private Information from Language Models
This addresses a critical privacy issue for users of AI systems by demonstrating a practical data extraction method, though it is incremental in building on existing extraction attacks.
The paper tackles the problem of privacy risks in large language models trained on private data by proposing a 'neural phishing' attack that extracts sensitive information like credit card numbers with success rates up to 50%.
When large language models are trained on private data, it can be a significant privacy risk for them to memorize and regurgitate sensitive information. In this work, we propose a new practical data extraction attack that we call "neural phishing". This attack enables an adversary to target and extract sensitive or personally identifiable information (PII), e.g., credit card numbers, from a model trained on user data with upwards of 10% attack success rates, at times, as high as 50%. Our attack assumes only that an adversary can insert as few as 10s of benign-appearing sentences into the training dataset using only vague priors on the structure of the user data.