DGPO: Beyond Pairwise Preferences with Directional Consistent Groupwise Optimization
For LLM alignment, this method addresses the limitation of pairwise preferences by leveraging group-level supervision, yielding consistent improvements.
DGPO introduces a groupwise preference optimization method that improves directional consistency in LLMs, achieving up to 3.6% average accuracy gains across benchmarks.
Although Large Language Models (LLMs) have made remarkable progress, current preference optimization methods still struggle to align directional consistency while preserving reasoning diversity. To address this limitation, we propose Directional-Groupwise Preference Optimization (DGPO), a lightweight framework that aggregates supervision signals at the group level and explicitly models direction-aware alignment through multi-candidate comparisons. DGPO organizes forward and reverse question-answer instances into structured sets and optimizes a margin-based likelihood objective that separates coherent reasoning paths from inconsistent alternatives. This group-wise formulation captures richer relative information than pairwise objectives and reinforces consistency across diverse reasoning pathways. Empirical results show that our constructed reverse data yields a 3.2% average improvement across five benchmarks, while DGPO further delivers consistent gains across multiple datasets and model families, achieving average accuracy improvements of up to 3.6%.