Self-Evolution Fine-Tuning for Policy Optimization
This addresses the problem of costly and unstable alignment methods for LLMs, offering a more efficient approach, though it appears incremental as it builds on existing fine-tuning techniques.
The paper tackles the challenge of aligning large language models without requiring annotated samples by introducing self-evolution fine-tuning (SEFT), which uses an adaptive reviser to enhance low-quality responses and fine-tune the policy, achieving effectiveness on benchmarks like AlpacaEval 2.0 and MT-Bench.
The alignment of large language models (LLMs) is crucial not only for unlocking their potential in specific tasks but also for ensuring that responses meet human expectations and adhere to safety and ethical principles. Current alignment methodologies face considerable challenges. For instance, supervised fine-tuning (SFT) requires extensive, high-quality annotated samples, while reinforcement learning from human feedback (RLHF) is complex and often unstable. In this paper, we introduce self-evolution fine-tuning (SEFT) for policy optimization, with the aim of eliminating the need for annotated samples while retaining the stability and efficiency of SFT. SEFT first trains an adaptive reviser to elevate low-quality responses while maintaining high-quality ones. The reviser then gradually guides the policy's optimization by fine-tuning it with enhanced responses. One of the prominent features of this method is its ability to leverage unlimited amounts of unannotated data for policy optimization through supervised fine-tuning. Our experiments on AlpacaEval 2.0 and MT-Bench demonstrate the effectiveness of SEFT. We also provide a comprehensive analysis of its advantages over existing alignment techniques.