Why mask diffusion does not work
This addresses a key limitation in diffusion models for language generation, but it is incremental as it focuses on improving existing methods rather than introducing a new paradigm.
This paper tackles the problem of mask diffusion language models failing to achieve parallel generation and bidirectional attention, demonstrating inherent difficulties and proposing effective training and inference strategies.
The main advantages of diffusion language models over autoregressive (AR) models lie in their ability to support parallel generation and bidirectional attention, enabling a more controllable generation process. In recent years, open-source mask diffusion language models have emerged, most of which are based on a variant known as absorbing diffusion. However, this paper demonstrates why mask diffusion faces inherent difficulties in achieving parallel generation and bidirectional attention. We also propose the most effective training and inference strategies for mask diffusion.