Diffusion on the Probability Simplex
This addresses a fundamental challenge in generative modeling for discrete data, but it appears incremental as it adapts existing diffusion concepts to a new mathematical framework.
The paper tackles the tension between continuous diffusion processes and discrete data by proposing diffusion on the probability simplex, using a softmax function applied to an Ornstein-Uhlenbeck process, and finds it extends to the unit cube for bounded image generation.
Diffusion models learn to reverse the progressive noising of a data distribution to create a generative model. However, the desired continuous nature of the noising process can be at odds with discrete data. To deal with this tension between continuous and discrete objects, we propose a method of performing diffusion on the probability simplex. Using the probability simplex naturally creates an interpretation where points correspond to categorical probability distributions. Our method uses the softmax function applied to an Ornstein-Unlenbeck Process, a well-known stochastic differential equation. We find that our methodology also naturally extends to include diffusion on the unit cube which has applications for bounded image generation.