Debiasing Guidance for Discrete Diffusion with Sequential Monte Carlo
This work addresses a specific limitation in guidance methods for discrete diffusion models, offering an incremental improvement for researchers and practitioners in generative modeling.
The paper tackled the problem of targeting specific regions in discrete diffusion models by introducing a Sequential Monte Carlo algorithm that generates unbiased samples from a desired distribution, achieving strong control in text generation with low perplexity compared to existing methods.
Discrete diffusion models are a class of generative models that produce samples from an approximated data distribution within a discrete state space. Often, there is a need to target specific regions of the data distribution. Current guidance methods aim to sample from a distribution with mass proportional to $p_0(x_0) p(ζ|x_0)^α$ but fail to achieve this in practice. We introduce a Sequential Monte Carlo algorithm that generates unbiasedly from this target distribution, utilising the learnt unconditional and guided process. We validate our approach on low-dimensional distributions, controlled images and text generations. For text generation, our method provides strong control while maintaining low perplexity compared to guidance-based approaches.