SGPO: Self-Generated Preference Optimization based on Self-Improver
This addresses the challenge of distribution shift and limited applicability in LLM alignment for practical deployment, though it appears incremental as it builds on existing methods like DPO.
The paper tackles the problem of aligning large language models to human preferences without relying on human-annotated datasets, proposing SGPO, which uses a self-improving mechanism to generate preference data, resulting in significant performance improvements over DPO and baseline methods on benchmarks like AlpacaEval 2.0 and Arena-Hard.
Large language models (LLMs), despite their extensive pretraining on diverse datasets, require effective alignment to human preferences for practical and reliable deployment. Conventional alignment methods typically employ off-policy learning and depend on human-annotated datasets, which limits their broad applicability and introduces distribution shift issues during training. To address these challenges, we propose Self-Generated Preference Optimization based on Self-Improver (SGPO), an innovative alignment framework that leverages an on-policy self-improving mechanism. Specifically, the improver refines responses from a policy model to self-generate preference data for direct preference optimization (DPO) of the policy model. Here, the improver and policy are unified into a single model, and in order to generate higher-quality preference data, this self-improver learns to make incremental yet discernible improvements to the current responses by referencing supervised fine-tuning outputs. Experimental results on AlpacaEval 2.0 and Arena-Hard show that the proposed SGPO significantly improves performance over DPO and baseline self-improving methods without using external preference data.