Memorization in Language Models through the Lens of Intrinsic Dimension
This research addresses privacy and intellectual property concerns in language models by identifying a geometric factor that influences memorization, offering incremental insights into existing drivers like model scale and data duplication.
The study tackled the problem of unintended memorization in language models by investigating how Intrinsic Dimension (ID), a measure of sequence complexity in latent space, modulates memorization rates, finding that high-ID sequences are less likely to be memorized, especially in overparameterized models with sparse exposure.
Language Models (LMs) are prone to memorizing parts of their data during training and unintentionally emitting them at generation time, raising concerns about privacy leakage and disclosure of intellectual property. While previous research has identified properties such as context length, parameter size, and duplication frequency, as key drivers of unintended memorization, little is known about how the latent structure modulates this rate of memorization. We investigate the role of Intrinsic Dimension (ID), a geometric proxy for the structural complexity of a sequence in latent space, in modulating memorization. Our findings suggest that ID acts as a suppressive signal for memorization: compared to low-ID sequences, high-ID sequences are less likely to be memorized, particularly in overparameterized models and under sparse exposure. These findings highlight the interaction between scale, exposure, and complexity in shaping memorization.