Aligning Large Language Models via Fully Self-Synthetic Data
This provides a practical, cost-effective solution for aligning LLMs, though it is incremental as it builds on existing alignment methods by automating data generation.
The paper tackles the high cost of human or AI feedback for aligning large language models by introducing Self-Alignment Optimization (SAO), a fully self-synthetic framework that generates all training data internally, resulting in enhanced chat capabilities on benchmarks like AlpacaEval~2.0 while maintaining performance on tasks such as question-answering and math reasoning.
Traditional reinforcement learning from human feedback (RLHF) for large language models (LLMs) relies on expensive human-annotated datasets, while Reinforcement Learning from AI Feedback (RLAIF) also incurs significant costs, requiring the collection of diverse prompts and corresponding responses, often necessitating external reward models or proprietary models like GPT-4 to annotate preference pairs. In this work, we introduce Self-Alignment Optimization (SAO), a fully self-synthetic framework for LLM alignment, where all training data, including prompts (i.e., user queries), responses, and preferences, are generated by the model itself. Specifically, SAO first instructs the LLM to engage in persona role-play and generate diverse prompts and responses, which are then self-evaluated for preference optimization. Extensive experiments demonstrate that SAO effectively enhances the model's chat capabilities on standard benchmarks like AlpacaEval~2.0, while maintaining strong performance on downstream objective tasks (e.g., question-answering, math reasoning). Our work provides a practical solution for self-improvement in aligning LLMs, and the code for reproducing our results is available at: https://github.com/SJY8460/SAO.