PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation
This addresses the problem of multi-objective alignment for LLM developers and users, offering a more efficient and flexible solution, though it appears incremental as it builds on existing alignment methods like RLHF and DPO.
The paper tackled the challenge of aligning large language models with conflicting human objectives like helpfulness and harmlessness by proposing PEO, a framework that generates Pareto-optimal policies in a single training pass, achieving superior Pareto fronts and improved flexibility and computational efficiency compared to baselines.
The alignment of large language models with human values presents a critical challenge, particularly when balancing conflicting objectives like helpfulness and harmlessness. Existing approaches, such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), face notable limitations: RLHF suffers from instability and inefficiency in multi-objective optimization, while DPO lacks mechanisms for dynamic trade-offs. To address these challenges, we propose Post-Training Extrapolation Optimization (PEO), a novel and efficient framework for bi-factorial alignment. PEO generates a family of Pareto-optimal policies in a single training pass by leveraging a three-phase pipeline: (1) aspect-specific learning, (2) generalist initialization via interpolation, and (3) post-training optimization via extrapolation. PEO enables dynamic adaptation to diverse user preferences at inference time without retraining. Our comprehensive experiments across multiple LLMs demonstrate that PEO achieves superior Pareto fronts compared to baselines, offering improved flexibility and computational efficiency. Theoretical analyses further highlight PEO's capacity to overcome optimization bottlenecks, paving the way for scalable, personalized alignment.