Discrete Diffusion with Sample-Efficient Estimators for Conditionals
This work addresses the problem of efficient discrete generative modeling for researchers in machine learning, offering incremental improvements over prior methods.
The authors tackled generative modeling over discrete state spaces by proposing a discrete diffusion framework that uses sample-efficient estimators for single-site conditionals, outperforming existing methods on binary datasets with improvements in metrics like total variation and cross-correlations.
We study a discrete denoising diffusion framework that integrates a sample-efficient estimator of single-site conditionals with round-robin noising and denoising dynamics for generative modeling over discrete state spaces. Rather than approximating a discrete analog of a score function, our formulation treats single-site conditional probabilities as the fundamental objects that parameterize the reverse diffusion process. We employ a sample-efficient method known as Neural Interaction Screening Estimator (NeurISE) to estimate these conditionals in the diffusion dynamics. Controlled experiments on synthetic Ising models, MNIST, and scientific data sets produced by a D-Wave quantum annealer, synthetic Potts model and one-dimensional quantum systems demonstrate the proposed approach. On the binary data sets, these experiments demonstrate that the proposed approach outperforms popular existing methods including ratio-based approaches, achieving improved performance in total variation, cross-correlations, and kernel density estimation metrics.