Swordsman: Entropy-Driven Adaptive Block Partition for Efficient Diffusion Language Models
This addresses efficiency and quality issues in diffusion language models for natural language processing applications, representing an incremental improvement over existing block-wise decoding methods.
The paper tackles the problem of rigid block partitioning in diffusion language models, which fragments semantic constituents and reduces performance, by proposing Swordsman, an entropy-driven adaptive block-wise decoding framework that improves inference speed and quality, achieving state-of-the-art results in evaluations.
Block-wise decoding effectively improves the inference speed and quality in diffusion language models (DLMs) by combining inter-block sequential denoising and intra-block parallel unmasking. However, existing block-wise decoding methods typically partition blocks in a rigid and fixed manner, which inevitably fragments complete semantic or syntactic constituents, leading to suboptimal performance. Inspired by the entropy reduction hypothesis (ERH), we recognize that constituent boundaries offer greater opportunities for uncertainty reduction, which motivates us to employ entropy analysis for identifying constituent boundaries. Therefore, we propose Swordsman, an entropy-driven adaptive block-wise decoding framework for DLMs. Swordsman adaptively partitions blocks by identifying entropy shifts between adjacent tokens to better align with semantic or syntactic constituent boundaries. In addition, Swordsman dynamically adjusts unmasking thresholds conditioned on the real-time unmasking status within a block, further improving both efficiency and stability. As a training-free framework, supported by KV Cache, Swordsman demonstrates state-of-the-art performance across extensive evaluations.