Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs
This work addresses the issue of data leakage and memorization risks in LLMs for AI safety and security researchers, though it is incremental in building on existing memorization quantification approaches.
The paper tackles the problem of quantifying memorization in large language models (LLMs) by introducing a black-box prompt optimization method that uses an attacker LLM to uncover higher levels of memorization, resulting in a 23.7% higher overlap with training data compared to baseline methods.
In this paper, we introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent, compared to what is revealed by prompting the target model with the training data directly, which is the dominant approach of quantifying memorization in LLMs. We use an iterative rejection-sampling optimization process to find instruction-based prompts with two main characteristics: (1) minimal overlap with the training data to avoid presenting the solution directly to the model, and (2) maximal overlap between the victim model's output and the training data, aiming to induce the victim to spit out training data. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that (1) instruction-tuned models can expose pre-training data as much as their base-models, if not more so, (2) contexts other than the original training data can lead to leakage, and (3) using instructions proposed by other LLMs can open a new avenue of automated attacks that we should further study and explore. The code can be found at https://github.com/Alymostafa/Instruction_based_attack .