Offline Preference Optimization via Maximum Marginal Likelihood Estimation
This work addresses the complexity and instability in aligning LLMs with human preferences, offering a simpler method that could benefit researchers and practitioners in AI alignment, though it appears incremental as it builds on existing preference optimization paradigms.
The paper tackles the problem of aligning Large Language Models with human preferences by proposing a simpler approach called Maximum Marginal Likelihood Preference Optimization (MMPO), which recasts alignment through Maximum Marginal Likelihood estimation and forgoes the need for an explicit reward model and entropy maximization. The result shows that MMPO is more stable with respect to hyperparameters and achieves competitive or superior preference alignment while better preserving the base model's general language capabilities across models from 135M to 8B parameters.
Aligning Large Language Models (LLMs) with human preferences is crucial, but standard methods like Reinforcement Learning from Human Feedback (RLHF) are often complex and unstable. In this work, we propose a new, simpler approach that recasts alignment through the lens of Maximum Marginal Likelihood (MML) estimation. Our new MML based Preference Optimization (MMPO) maximizes the marginal log-likelihood of a preferred text output, using the preference pair as samples for approximation, and forgoes the need for both an explicit reward model and entropy maximization. We theoretically demonstrate that MMPO implicitly performs preference optimization, producing a weighted gradient that naturally up-weights chosen responses over rejected ones. Across models ranging from 135M to 8B parameters, we empirically show that MMPO: 1) is more stable with respect to the hyperparameter $β$ compared to alternative baselines, and 2) achieves competitive or superior preference alignment while better preserving the base model's general language capabilities. Through a series of ablation experiments, we show that this improved performance is indeed attributable to MMPO's implicit preference optimization within the gradient updates.