Reinforcement Learning-based Knowledge Distillation with LLM-as-a-Judge
This work addresses the need for supervision in RL fine-tuning for language models, offering a method to leverage unlabeled data, though it appears incremental as it builds on existing RL and knowledge distillation techniques.
The authors tackled the problem of requiring ground truth labels for reinforcement learning in language models by proposing a framework that uses an LLM as a judge to provide rewards from unlabeled data, enabling label-free knowledge distillation and achieving substantial performance gains on math reasoning benchmarks.
Reinforcement Learning (RL) has been shown to substantially improve the reasoning capability of small and large language models (LLMs), but existing approaches typically rely on verifiable rewards, hence ground truth labels. We propose an RL framework that uses rewards from an LLM that acts as a judge evaluating model outputs over large amounts of unlabeled data, enabling label-free knowledge distillation and replacing the need of ground truth supervision. Notably, the judge operates with a single-token output, making reward computation efficient. When combined with verifiable rewards, our approach yields substantial performance gains across math reasoning benchmarks. These results suggest that LLM-based evaluators can produce effective training signals for RL fine-tuning.