GATES: Self-Distillation under Privileged Context with Consensus Gating
This addresses the challenge of training models when supervision is unreliable, which is incremental but important for applications like document-grounded QA.
The paper tackles the problem of self-distillation in unreliable supervision settings, such as document-grounded question answering without ground truth labels, by using tutor consensus to gate learning and distilling knowledge from full reasoning trajectories, resulting in improvements from 46.0% to 62.0% in held-out accuracy and from 20.2% to 35.4% on public benchmarks.
We study self-distillation in settings where supervision is unreliable: there are no ground truth labels, verifiable rewards, or external graders to evaluate answers. We focus on document-grounded question answering with asymmetric context, where a single model serves as both tutor (with access to a relevant source document during training) and student (answering from the question alone at test time). Rather than assuming tutor correctness, we derive supervision online from tutor consensus by sampling multiple document-grounded reasoning traces and using agreement to gate learning. Conditioned on this reliability signal, we distill knowledge through full tutor reasoning trajectories (not just final answers), providing a dense and stable learning signal. Empirically, this consensus-gated trajectory distillation substantially improves transfer to the document-free student. Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.