Score-based Continuous-time Discrete Diffusion Models
This work addresses a key limitation in diffusion models for discrete spaces, enabling applications in domains like music and image generation, though it is incremental in extending existing continuous methods.
The paper tackled the problem of adapting score-based diffusion models to discrete data by introducing a stochastic jump process for categorical variables, achieving effective performance on synthetic and real-world music and image benchmarks.
Score-based modeling through stochastic differential equations (SDEs) has provided a new perspective on diffusion models, and demonstrated superior performance on continuous data. However, the gradient of the log-likelihood function, i.e., the score function, is not properly defined for discrete spaces. This makes it non-trivial to adapt \textcolor{\cdiff}{the score-based modeling} to categorical data. In this paper, we extend diffusion models to discrete variables by introducing a stochastic jump process where the reverse process denoises via a continuous-time Markov chain. This formulation admits an analytical simulation during backward sampling. To learn the reverse process, we extend score matching to general categorical data and show that an unbiased estimator can be obtained via simple matching of the conditional marginal distributions. We demonstrate the effectiveness of the proposed method on a set of synthetic and real-world music and image benchmarks.