Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects
This addresses privacy and copyright concerns for users of language models by revealing that DLMs may offer reduced memorization risks, though it is incremental as it builds on prior work on ARMs.
The paper tackled the problem of characterizing memorization in diffusion language models (DLMs), which had been unexplored compared to autoregressive models, and found that increasing sampling resolution strictly increases exact training data extraction, with DLMs showing substantially lower leakage of personally identifiable information (PII) than ARMs in experiments.
Autoregressive language models (ARMs) have been shown to memorize and occasionally reproduce training data verbatim, raising concerns about privacy and copyright liability. Diffusion language models (DLMs) have recently emerged as a competitive alternative, yet their memorization behavior remains largely unexplored due to fundamental differences in generation dynamics. To address this gap, we present a systematic theoretical and empirical characterization of memorization in DLMs. We propose a generalized probabilistic extraction framework that unifies prefix-conditioned decoding and diffusion-based generation under arbitrary masking patterns and stochastic sampling trajectories. Theorem 4.3 establishes a monotonic relationship between sampling resolution and memorization: increasing resolution strictly increases the probability of exact training data extraction, implying that autoregressive decoding corresponds to a limiting case of diffusion-based generation by setting the sampling resolution maximal. Extensive experiments across model scales and sampling strategies validate our theoretical predictions. Under aligned prefix-conditioned evaluations, we further demonstrate that DLMs exhibit substantially lower memorization-based leakage of personally identifiable information (PII) compared to ARMs.