PIPA: Preference Alignment as Prior-Informed Statistical Estimation
This provides a unified approach for aligning language models with human preferences, which is incremental as it builds on existing methods like DPO and KTO.
The paper tackles the lack of a unified framework for offline preference alignment in language models by introducing PIPA, a probabilistic method that formulates alignment as maximum likelihood estimation with prior constraints, resulting in 3-10% performance improvements on GSM8K and MATH benchmarks without extra computational costs.
Offline preference alignment for language models such as Direct Preference Optimization (DPO) is favored for its effectiveness and simplicity, eliminating the need for costly reinforcement learning. Various offline algorithms have been developed for different data settings, yet they lack a unified understanding. In this study, we introduce Pior-Informed Preference Alignment (PIPA), a unified, RL-free probabilistic framework that formulates language model preference alignment as a Maximum Likelihood Estimation (MLE) problem with prior constraints. This method effectively accommodates both paired and unpaired data, as well as answer and step-level annotations. We illustrate that DPO and KTO are special cases with different prior constraints within our framework. By integrating different types of prior information, we developed two variations of PIPA: PIPA-M and PIPA-N. Both algorithms demonstrate a $3\sim10\%$ performance enhancement on the GSM8K and MATH benchmarks across all configurations, achieving these gains without additional training or computational costs compared to existing algorithms.