Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
This work addresses the problem of improving alignment efficiency and effectiveness for LLM developers and researchers by moving beyond pairwise methods, though it is incremental as it builds on existing preference optimization techniques.
The paper tackles the limitation of pairwise preference optimization in aligning large language models by proposing Ranked Choice Preference Optimization (RCPO), a framework that incorporates richer human feedback like multiwise comparisons and top-k rankings, and shows it consistently outperforms baselines on benchmarks such as AlpacaEval 2 and Arena-Hard with models like Llama-3-8B-Instruct and Gemma-2-9B-it.
Alignment of large language models (LLMs) has predominantly relied on pairwise preference optimization, where annotators select the better of two responses to a prompt. While simple, this approach overlooks the opportunity to learn from richer forms of human feedback, such as multiwise comparisons and top-$k$ rankings. We propose Ranked Choice Preference Optimization (RCPO), a unified framework that bridges preference optimization with (ranked) choice modeling via maximum likelihood estimation. The framework is flexible, supporting both utility-based and rank-based choice models. It subsumes several existing pairwise methods (e.g., DPO, SimPO), while providing principled training objectives for richer feedback formats. We instantiate this framework with two representative ranked choice models (Multinomial Logit and Mallows-RMJ). Empirical studies on Llama-3-8B-Instruct and Gemma-2-9B-it across AlpacaEval 2 and Arena-Hard benchmarks show that RCPO consistently outperforms competitive baselines. RCPO shows how directly leveraging ranked preference data, combined with the right choice models, yields more effective alignment. It offers a versatile and extensible foundation for incorporating (ranked) choice modeling into LLM training.