Triplets Better Than Pairs: Towards Stable and Effective Self-Play Fine-Tuning for LLMs
This addresses the problem of fine-tuning LLMs with scarce annotated data, offering a more stable and effective method, though it is incremental as it builds upon existing self-play approaches.
The paper tackles the instability and misalignment issues in self-play fine-tuning for large language models by proposing T-SPIN, which incorporates historical advantages and entropy constraints to stabilize optimization and eliminate training-generation discrepancies. Empirical results show T-SPIN outperforms SPIN, achieves stable evolution, and matches or exceeds supervised fine-tuning performance with only 25% of samples.
Recently, self-play fine-tuning (SPIN) has been proposed to adapt large language models to downstream applications with scarce expert-annotated data, by iteratively generating synthetic responses from the model itself. However, SPIN is designed to optimize the current reward advantages of annotated responses over synthetic responses at hand, which may gradually vanish during iterations, leading to unstable optimization. Moreover, the utilization of reference policy induces a misalignment issue between the reward formulation for training and the metric for generation. To address these limitations, we propose a novel Triplet-based Self-Play fIne-tuNing (T-SPIN) method that integrates two key designs. First, beyond current advantages, T-SPIN additionally incorporates historical advantages between iteratively generated responses and proto-synthetic responses produced by the initial policy. Even if the current advantages diminish, historical advantages remain effective, stabilizing the overall optimization. Second, T-SPIN introduces the entropy constraint into the self-play framework, which is theoretically justified to support reference-free fine-tuning, eliminating the training-generation discrepancy. Empirical results on various tasks demonstrate not only the superior performance of T-SPIN over SPIN, but also its stable evolution during iterations. Remarkably, compared to supervised fine-tuning, T-SPIN achieves comparable or even better performance with only 25% samples, highlighting its effectiveness when faced with scarce annotated data.