EviSD: Evidence-Conditioned Self-Distillation for Search-Augmented Agents
This work improves the training efficiency and performance of search-augmented language agents for question answering, a domain-specific problem, by addressing the credit assignment bottleneck in outcome-based RL.
EviSD introduces an evidence-conditioned self-distillation framework that uses instance-level supporting evidence as privileged information to improve credit assignment in outcome-based reinforcement learning for search-augmented language agents. Across seven QA benchmarks and three backbones, EviSD achieves the highest macro-average Exact Match, outperforming the strongest baselines by 1.3-2.3 points while modulating only 6.7-15.1% of response tokens.
Outcome-based reinforcement learning enables search-augmented language agents to learn from verifiable final answers, but its trajectory-level credit cannot distinguish the contributions of individual actions in a multi-turn search process. We propose EviSD, an evidence-conditioned self-distillation framework that uses instance-level supporting evidence as privileged information for search actions and golden answers as complementary privilege for answer actions. During training, the student samples actions from the original context, while the same model re-scores them as a privileged teacher under an action-aligned context. EviSD converts the detached teacher--student gap into a bounded correction to the outcome-derived GRPO advantage and applies it only to generated action spans. This design localizes privileged guidance while preserving the update direction determined by the outcome reward, without an auxiliary distillation objective or any change at inference time. Across seven question-answering benchmarks and three backbones spanning model scales and generations, EviSD achieves the highest macro-average Exact Match in all evaluated settings, outperforming the strongest compared methods by 1.3--2.3 points while modulating only 6.7%--15.1% of response tokens. Code is available at https://github.com/JiananXie/EviSD.