Mind the Gap: Structure-Aware Consistency in Preference Learning
For researchers and practitioners aligning LLMs, this work addresses a fundamental theoretical flaw in popular preference learning methods and offers a principled fix with improved consistency guarantees.
The paper identifies that standard surrogate losses used in preference learning for LLMs, such as DPO, are theoretically inconsistent for neural networks, leading to vacuous generalization guarantees. It proposes a structure-aware margin-shifted ranking framework (SA-DPO) that adapts margins based on semantic distance between responses, and shows that heavy-tailed surrogates like Polynomial Hinge provide better consistency guarantees for capacity-bounded models.
Preference learning has become the foundation of aligning Large Language Models (LLMs) with human intent. Popular methods, such as Direct Preference Optimization (DPO), minimize surrogate losses as proxies for the intractable pairwise ranking loss. However, we demonstrate that for the equicontinuous hypothesis sets typical of neural networks, these standard surrogates are theoretically inconsistent, yielding vacuous generalization guarantees. To resolve this, we formulate LLM alignment within a margin-shifted ranking framework. We derive rigorous $H$-consistency bounds that depend on enforcing a separation margin $γ$. Crucially, we extend this to Structure-Aware $H$-consistency, introducing a novel objective (SA-DPO) that adapts the margin based on the semantic distance between responses to handle synonyms and hard pairs. Finally, we analyze the trade-off between consistency and model limitations via the Margin-Capacity Profile, proving that heavy-tailed surrogates (such as the Polynomial Hinge family) offer superior consistency guarantees for capacity-bounded models compared to the standard logistic loss used in DPO.