Adaptive Guidance for Retrieval-Augmented Masked Diffusion Models
This work addresses a specific challenge in integrating external knowledge into diffusion models for knowledge-intensive tasks, representing an incremental advancement in retrieval-augmented generation methods.
The paper tackles the problem of retrieval-prior conflicts in retrieval-augmented generation for diffusion-based language models, where noisy or inconsistent retrieved context degrades quality, and proposes ARAM, a training-free adaptive guidance framework that dynamically adjusts guidance based on signal-to-noise ratio, resulting in improved QA performance on multiple benchmarks.
Retrieval-Augmented Generation (RAG) improves factual grounding by incorporating external knowledge into language model generation. However, when retrieved context is noisy, unreliable, or inconsistent with the model's parametric knowledge, it introduces retrieval-prior conflicts that can degrade generation quality. While this problem has been studied in autoregressive language models, it remains largely unexplored in diffusion-based language models, where the iterative denoising process introduces unique challenges for integrating retrieved context. In this work, we propose Adaptive Retrieval-Augmented Masked Diffusion (ARAM), a training-free adaptive guidance framework for Masked Diffusion Models (MDMs) in RAG settings. ARAM dynamically calibrates the guidance scale during denoising according to the Signal-to-Noise Ratio (SNR) of the distributional shift induced by retrieved context. Intuitively, the model strengthens guidance when the retrieved context provides reliable corrective evidence and suppresses it when the contextual signal is noisy or non-supportive. Extensive experiments on multiple knowledge-intensive QA benchmarks show that ARAM improves overall QA performance over competitive RAG baselines.