D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decoding
This addresses the need for better diversity control in parallel decoding for discrete diffusion models, which is an incremental advancement over existing techniques.
The paper tackles the problem of limited control over in-batch diversity in decoding for discrete diffusion models in text generation, introducing D5P4, a method that improves diversity while maintaining competitive quality, as shown in experiments on free-form generation and question answering.
Discrete diffusion models are promising alternatives to autoregressive approaches for text generation, yet their decoding methods remain under-studied. Standard decoding methods for autoregressive models, such as beam search, do not directly apply to iterative denoising, and existing diffusion decoding techniques provide limited control over in-batch diversity. To bridge this gap, we introduce a generalized beam-search framework for discrete diffusion that generates candidates in parallel and supports modular beam-selection objectives. As a diversity-focused instantiation, we propose D5P4, which formulates the selection step as MAP inference over a Determinantal Point Process. Leveraging a scalable greedy solver, D5P4 maintains multi-GPU compatibility and enables an explicit trade-off between model probability and target diversity with near-zero compute overhead. Experiments on free-form generation and question answering demonstrate that D5P4 improves diversity over strong baselines while maintaining competitive generation quality.