From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons
Enables efficient reuse of pre-trained AR models for diffusion generation, solving a key structural mismatch problem for text generation practitioners.
FLUID adapts autoregressive LLMs to diffusion-based parallel text generation by enforcing causal attention alignment and using entropy-driven denoising, achieving SOTA performance with orders-of-magnitude reduction in training cost.
Diffusion models promise efficient parallel text generation but rely on bidirectional attention, creating a structural mismatch with pre-trained Autoregressive (AR) models. This incompatibility precludes reusing robust AR priors, necessitating prohibitive pre-training from scratch. To bridge this gap, we propose FLUID, a framework that efficiently adapts AR backbones to the diffusion paradigm. By enforcing Strictly Causal Alignment, FLUID enables seamless initialization from standard GPT-style checkpoints, circumventing the need for massive pre-training. Furthermore, we introduce Elastic Horizons, an entropy-driven mechanism that dynamically modulates denoising strides based on local information density rather than fixed schedules. Experiments demonstrate that FLUID achieves state-of-the-art performance while reducing training costs by orders of magnitude, effectively reconciling established AR foundations with efficient parallel generation. Our code is available at https://github.com/Oli-lab-nun/FLUID/tree/main.