Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models
This addresses the lack of transparency in commercial LLMs, enabling oversight for issues like copyright infringement and supporting research on data contamination, though it is incremental as it builds on existing memorization probe techniques.
The authors tackled the problem of identifying training data in proprietary large language models (LLMs) without access to model internals, by using information-guided probes based on high-surprisal text passages, and demonstrated that this method can identify a surprising number of memorized texts.
High-quality training data has proven crucial for developing performant large language models (LLMs). However, commercial LLM providers disclose few, if any, details about the data used for training. This lack of transparency creates multiple challenges: it limits external oversight and inspection of LLMs for issues such as copyright infringement, it undermines the agency of data authors, and it hinders scientific research on critical issues such as data contamination and data selection. How can we recover what training data is known to LLMs? In this work, we demonstrate a new method to identify training data known to proprietary LLMs like GPT-4 without requiring any access to model weights or token probabilities, by using information-guided probes. Our work builds on a key observation: text passages with high surprisal are good search material for memorization probes. By evaluating a model's ability to successfully reconstruct high-surprisal tokens in text, we can identify a surprising number of texts memorized by LLMs.