Deconstructing Generative Diversity: An Information Bottleneck Analysis of Discrete Latent Generative Models
This work addresses the challenge of understanding and controlling generative diversity in AI models, which is incremental as it builds on existing theories to provide a diagnostic tool.
The paper tackled the problem of varying generative diversity in discrete latent generative models like AR, MIM, and Diffusion by proposing an Information Bottleneck-based diagnostic framework to analyze and decompose diversity into path and execution components, revealing distinct strategies and enabling a new inference-time enhancement technique.
Generative diversity varies significantly across discrete latent generative models such as AR, MIM, and Diffusion. We propose a diagnostic framework, grounded in Information Bottleneck (IB) theory, to analyze the underlying strategies resolving this behavior. The framework models generation as a conflict between a 'Compression Pressure' - a drive to minimize overall codebook entropy - and a 'Diversity Pressure' - a drive to maximize conditional entropy given an input. We further decompose this diversity into two primary sources: 'Path Diversity', representing the choice of high-level generative strategies, and 'Execution Diversity', the randomness in executing a chosen strategy. To make this decomposition operational, we introduce three zero-shot, inference-time interventions that directly perturb the latent generative process and reveal how models allocate and express diversity. Application of this probe-based framework to representative AR, MIM, and Diffusion systems reveals three distinct strategies: "Diversity-Prioritized" (MIM), "Compression-Prioritized" (AR), and "Decoupled" (Diffusion). Our analysis provides a principled explanation for their behavioral differences and informs a novel inference-time diversity enhancement technique.