Self-Steering Optimization: Autonomous Preference Optimization for Large Language Models
This addresses the need for scalable, automated alignment techniques in AI by providing a quality control mechanism to improve iterative optimization, though it appears incremental as it builds on existing automated alignment methods.
The paper tackles the problem of generating high-quality preference data for aligning large language models by introducing Self-Steering Optimization (SSO), an algorithm that autonomously produces accurate and on-policy data, resulting in consistent outperformance over baselines in human preference alignment and reward optimization across benchmarks like Llama 3 and Qwen 2.
The key to effective alignment lies in high-quality preference data. Recent research has focused on automated alignment, which involves developing alignment systems with minimal human intervention. However, prior research has predominantly focused on developing data generation methods, while insufficient attention has been paid to quality control mechanisms, which often produce inaccurate and unhelpful data, leading to unpredictable benefits during iterative optimization. In this paper, we present Self-Steering Optimization ($SSO$), an algorithm that autonomously generates high-quality preference data, eliminating manual annotation requirements. $SSO$ employs a specialized optimization objective to build a data generator from the policy model itself, which is used to produce accurate and on-policy data. We demonstrate $SSO$'s effectiveness through comprehensive experiments on two series of models: Llama 3 and Qwen 2. Our evaluation across diverse benchmarks shows that $SSO$ consistently outperforms baselines in human preference alignment and reward optimization. Further analysis validates $SSO$ as a scalable framework for preference optimization, benefiting the advancement in automated alignment techniques.