Entropy-Memorization Law: Evaluating Memorization Difficulty of Data in LLMs
This work addresses the challenge of understanding and controlling data memorization in LLMs, which is crucial for privacy and security, but it is incremental as it builds on existing memorization research with a new empirical law.
The paper tackles the problem of characterizing memorization difficulty in Large Language Models (LLMs) by proposing the Entropy-Memorization Law, which shows a linear correlation between data entropy and memorization scores, and applies this to distinguish training from testing data for Dataset Inference.
Large Language Models (LLMs) are known to memorize portions of their training data, sometimes reproducing content verbatim when prompted appropriately. In this work, we investigate a fundamental yet under-explored question in the domain of memorization: How to characterize memorization difficulty of training data in LLMs? Through empirical experiments on OLMo, a family of open models, we present the Entropy-Memorization Law. It suggests that data entropy is linearly correlated with memorization score. Moreover, in a case study of memorizing highly randomized strings, or "gibberish", we observe that such sequences, despite their apparent randomness, exhibit unexpectedly low empirical entropy compared to the broader training corpus. Adopting the same strategy to discover Entropy-Memorization Law, we derive a simple yet effective approach to distinguish training and testing data, enabling Dataset Inference (DI).