Rainbow Padding: Mitigating Early Termination in Instruction-Tuned Diffusion LLMs
This addresses a critical vulnerability in instruction-tuned diffusion LLMs that affects their practical deployment by preventing early termination, though it is an incremental solution focused on a specific issue.
The paper tackles the problem of early termination in instruction-tuned diffusion LLMs, known as <eos> overflow, by introducing Rainbow Padding, which replaces repeated <eos> placeholders with a cycle of distinct padding tokens, resulting in substantial improvements in length robustness and output quality with as few as seven padding tokens.
Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive models, offering flexible generation orders and strong performance on complex reasoning tasks. However, instruction-tuned dLLMs exhibit a critical vulnerability we term \texttt{<eos>} overflow: as allocated sequence length increases, responses paradoxically become shorter, collapsing into early termination or degenerating into streams of \texttt{<eos>} tokens. Although noticed in practice, this issue has not been systematically analyzed. We trace its root cause to the dual role of \texttt{<eos>} as both termination and padding, which concentrates probability mass on \texttt{<eos>} at later positions and propagates backward to trigger early termination. To address this, we introduce Rainbow Padding, a simple remedy that replaces repeated \texttt{<eos>} placeholders with a repeating cycle of distinct padding tokens, distributing probability mass and breaking \texttt{<eos>} dominance. Experiments show that Rainbow Padding substantially improves length robustness and output quality, with as few as seven padding tokens sufficient to prevent early termination. Moreover, the method integrates efficiently into existing instruction-tuned models: LoRA fine-tuning for a single epoch on minimal data yields significant improvements, making this solution highly practical. The code is publicly available at https://github.com/quasar529/rainbow-padding.