CoRe: Context-Robust Remasking for Diffusion Language Models
This addresses decoding inefficiencies in diffusion language models for NLP tasks, representing an incremental improvement over existing revision strategies.
The paper tackles the problem of context rigidity in Masked Diffusion Models, where early token predictions with high confidence can misguide subsequent generation due to lack of full context, by proposing CoRe, a training-free inference-time revision framework that identifies context-brittle tokens through sensitivity to masked-context perturbations, resulting in improvements such as a 9.2 percentage point gain on the MBPP benchmark.
Standard decoding in Masked Diffusion Models (MDMs) is hindered by context rigidity: tokens are retained based on transient high confidence, often ignoring that early predictions lack full context. This creates cascade effects where initial inconsistencies misguide the remaining generation. Existing revision strategies attempt to mitigate this by relying on static confidence scores, but these signals are inherently myopic; inconsistent tokens can appear confident to the model itself. We propose Context-Robust Remasking (CoRe), a training-free framework for inference-time revision. Rather than trusting static token probabilities, CoRe identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations. We formalize revision as a robust optimization objective over context shifts and efficiently approximate this objective to prioritize unstable tokens for revision. On LLaDA-8B-Base, CoRe delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.