Aligning Large Language Models by On-Policy Self-Judgment
This addresses the trade-off in alignment methods for large language models, offering a more efficient approach for researchers and practitioners, though it appears incremental as it builds on existing alignment techniques.
The paper tackles the problem of aligning large language models with human preferences without requiring a separate reward model for on-policy learning, resulting in a parameter-efficient framework that outperforms baselines in preference benchmarks.
Existing approaches for aligning large language models with human preferences face a trade-off that requires a separate reward model (RM) for on-policy learning. In this paper, we present a novel alignment framework, SELF-JUDGE that (1) does on-policy learning and 2) is parameter efficient, as it does not require an additional RM for evaluating the samples for on-policy learning. To this end, we propose Judge-augmented Supervised Fine-Tuning (JSFT) to train a single model to act as both a policy and a judge. Specifically, we view the pairwise judgment task, choosing the better response from a response pair, as a special case of the instruction-following task. The resulting model can judge preferences of on-the-fly responses from current policy initialized from itself. Experimental results show the efficacy of SELF-JUDGE, outperforming baselines in preference benchmarks. We also show that the rejecting sampling by itself can improve performance further without an additional evaluator.