Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones
This work addresses a bottleneck in aligning diffusion large language models for complex reasoning tasks, offering a lightweight method to improve efficiency without extra compute, though it is incremental as it builds on existing RL frameworks.
The paper tackled the problem of inefficient reinforcement learning in diffusion large language models by showing that reasoning errors are concentrated in specific 'confusion zones' with high uncertainty, and proposed Adaptive Trajectory Policy Optimization (ATPO) to reallocate gradients to these steps, achieving substantial gains in reasoning accuracy and training stability across benchmarks.
Diffusion Large Language Models (dLLMs) are rapidly emerging alongside autoregressive models as a powerful paradigm for complex reasoning, with reinforcement learning increasingly used for downstream alignment. Existing trajectory-based RL methods uniformly allocate policy gradients across denoising steps, implicitly treating all steps as equally important. We challenge this assumption by analyzing trajectories with several step-level metrics: entropy-based uncertainty, Confidence-Margin (CM) uncertainty, and Rate of Entropy Change (RoEC). These reveal structured "zones of confusion": transient spikes in uncertainty and instability that strongly predict final success or failure, while most steps remain stable. We propose Adaptive Trajectory Policy Optimization (ATPO), a lightweight step-selection strategy that dynamically reallocates gradient updates to these high-leverage steps without changing the RL objective, rewards, or compute budget. Using a hybrid RoEC+CM rule, ATPO delivers substantial gains in reasoning accuracy and training stability across benchmarks, showing that exploiting trajectory dynamics is key to advancing dLLM RL.