Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
This work addresses the challenge of efficient model distillation for AI practitioners, offering a scalable approach that reduces reliance on external supervision, though it is incremental in improving existing distillation techniques.
The paper tackles the problem of distilling large language models by proposing a method that transfers both data and reward signals, enabling student models to surpass teacher performance on benchmarks like GSM8K and MMLU-PRO.
Distilling large language models (LLMs) typically involves transferring the teacher model's responses through supervised fine-tuning (SFT). However, this approach neglects the potential to distill both data (output content) and reward signals (quality evaluations). Extracting reliable reward signals directly from teacher models is challenging, as LLMs are optimized for generation rather than evaluation, often resulting in biased or inconsistent assessments. To address this limitation, we propose a novel distillation pipeline that transfers both responses and rewards. Our method generates pseudo-rewards through a self-supervised mechanism that leverages the inherent structure of both teacher and student responses, enabling reward learning without explicit external evaluation. The reward model subsequently guides reinforcement learning (RL), allowing iterative refinement of the student model after an SFT warm-up phase. Experiments on GSM8K and MMLU-PRO demonstrate that our method consistently outperforms traditional SFT-based approaches, enabling student models to surpass the performance of their teachers. This work highlights the potential for scalable, efficient distillation through structured self-supervised reward learning, reducing dependence on external reward supervision.