Deterministic Discrete Denoising
This work addresses the need for more efficient and higher-quality generative modeling in discrete domains, though it appears incremental as it extends deterministic methods from continuous to discrete spaces.
The paper tackled the problem of stochastic denoising in discrete-state diffusion models by proposing a deterministic algorithm based on Markov chains and herding, resulting in consistent improvements in efficiency and sample quality for text and image generation tasks.
We propose a deterministic denoising algorithm for discrete-state diffusion models based on Markov chains. The generative reverse process is derandomized by introducing a variant of the herding algorithm with weakly chaotic dynamics, which induces deterministic discrete state transitions. Our approach is a direct replacement for the stochastic denoising process, requiring neither retraining nor continuous state embeddings. We demonstrate consistent improvements in both efficiency and sample quality on text and image generation tasks. Thus, this simple derandomization approach is expected to enhance the significance of discrete diffusion in generative modeling. Furthermore, our results reveal that deterministic reverse processes, well established in continuous diffusion, can also be effective in discrete state spaces.