Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
This work addresses the problem of improving efficiency and flexibility in language generation for AI researchers and practitioners, representing a novel method rather than an incremental improvement.
The paper tackles the limitations of diffusion language models in likelihood modeling and fixed-length generation by introducing block diffusion models that interpolate between diffusion and autoregressive approaches, achieving state-of-the-art performance among diffusion models on language modeling benchmarks and enabling arbitrary-length sequence generation.
Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms