Decoding Large Language Diffusion Models with Foreseeing Movement
This addresses a critical bottleneck in parallelized inference for diffusion-based language models, though it appears to be an incremental improvement over existing heuristic methods.
The paper tackles the challenge of decoding order sensitivity in Large Language Diffusion Models by proposing Foreseeing Decoding Method (FDM), which integrates local and global considerations to optimize token selection, and its accelerated variant FDM-A that restricts deep exploration to critical steps, achieving superior efficiency-performance trade-offs across diverse benchmarks.
Large Language Diffusion Models (LLDMs) benefit from a flexible decoding mechanism that enables parallelized inference and controllable generations over autoregressive models. Yet such flexibility introduces a critical challenge: inference performance becomes highly sensitive to the decoding order of tokens. Existing heuristic methods, however, focus mainly on local effects while overlooking long-term impacts. To address this limitation, we propose the Foreseeing Decoding Method (FDM), a novel approach that integrates both local and global considerations to unlock the full potential, employing a search-based strategy to enable effective optimization in discrete spaces. Furthermore, by analyzing the consistency of chosen tokens in the full decoding process, we develop a variant, FDM with Acceleration (FDM-A), which restricts deep exploration to critical steps identified as the exploration and balance circumantences. Extensive experiments across diverse benchmarks and model architectures validate the scalability of FDM and demonstrate the superior efficiency-performance trade-off achieved by FDM-A. Our work might potentially provide a principled step toward more powerful decoding methods for LLDMs.