Continuous diffusion for categorical data
This work addresses a bottleneck in generative modeling for discrete data, offering a novel method that preserves the benefits of continuous diffusion.
The authors tackled the problem of applying continuous diffusion models to categorical data like language, proposing the CDCD framework and demonstrating its efficacy on several language modeling tasks.
Diffusion models have quickly become the go-to paradigm for generative modelling of perceptual signals (such as images and sound) through iterative refinement. Their success hinges on the fact that the underlying physical phenomena are continuous. For inherently discrete and categorical data such as language, various diffusion-inspired alternatives have been proposed. However, the continuous nature of diffusion models conveys many benefits, and in this work we endeavour to preserve it. We propose CDCD, a framework for modelling categorical data with diffusion models that are continuous both in time and input space. We demonstrate its efficacy on several language modelling tasks.