ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization
This addresses the problem of overoptimization and computational inefficiency in aligning LLMs with human preferences, though it appears incremental as it builds on existing Direct Alignment Algorithms.
The paper tackles the problem of inefficient preference optimization in large language models by introducing ConfPO, which selectively optimizes preference-critical tokens based on policy confidence rather than uniformly adjusting all tokens. Experimental results show ConfPO consistently outperforms uniform methods on benchmarks like AlpacaEval 2 and Arena-Hard with zero additional computational overhead.
We introduce ConfPO, a method for preference learning in Large Language Models (LLMs) that identifies and optimizes preference-critical tokens based solely on the training policy's confidence, without requiring any auxiliary models or compute. Unlike prior Direct Alignment Algorithms (DAAs) such as Direct Preference Optimization (DPO), which uniformly adjust all token probabilities regardless of their relevance to preference, ConfPO focuses optimization on the most impactful tokens. This targeted approach improves alignment quality while mitigating overoptimization (i.e., reward hacking) by using the KL divergence budget more efficiently. In contrast to recent token-level methods that rely on credit-assignment models or AI annotators, raising concerns about scalability and reliability, ConfPO is simple, lightweight, and model-free. Experimental results on challenging alignment benchmarks, including AlpacaEval 2 and Arena-Hard, demonstrate that ConfPO consistently outperforms uniform DAAs across various LLMs, delivering better alignment with zero additional computational overhead.