Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI Models
This addresses data privacy and security concerns for users and developers of AI models by identifying hidden memorization risks that could lead to unintended data leakage.
The paper tackled the problem of data leakage and memorization in frontier AI models by investigating how memorization evolves during training, revealing that sequences can be 'uncovered' as 'latent memorization' even without repeated encounters, and developed a diagnostic test using cross entropy loss to detect such sequences with high accuracy.
Frontier AI systems are making transformative impacts across society, but such benefits are not without costs: models trained on web-scale datasets containing personal and private data raise profound concerns about data privacy and security. Language models are trained on extensive corpora including potentially sensitive or proprietary information, and the risk of data leakage - where the model response reveals pieces of such information - remains inadequately understood. Prior work has investigated what factors drive memorization and have identified that sequence complexity and the number of repetitions drive memorization. Here, we focus on the evolution of memorization over training. We begin by reproducing findings that the probability of memorizing a sequence scales logarithmically with the number of times it is present in the data. We next show that sequences which are apparently not memorized after the first encounter can be "uncovered" throughout the course of training even without subsequent encounters, a phenomenon we term "latent memorization". The presence of latent memorization presents a challenge for data privacy as memorized sequences may be hidden at the final checkpoint of the model but remain easily recoverable. To this end, we develop a diagnostic test relying on the cross entropy loss to uncover latent memorized sequences with high accuracy.