Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization
This work addresses the problem of efficient and effective preference optimization for large language models, offering an incremental improvement over existing methods.
The paper tackles the challenge of post-training alignment in large language models by introducing a selective alignment strategy that prioritizes high-impact tokens within preference pairs, reducing computational overhead and enhancing alignment fidelity. Experiments on benchmarks like Arena-Hard and MT-Bench show that their Selective-DPO method outperforms standard DPO and distillation-based baselines.
Post-training alignment of large language models (LLMs) is a critical challenge, as not all tokens contribute equally to model performance. This paper introduces a selective alignment strategy that prioritizes high-impact tokens within preference pairs, leveraging token-level log-probability differences between the current policy and a reference model. By focusing on these informative tokens, our approach reduces computational overhead and enhances alignment fidelity. We further explore the role of reference model quality, demonstrating that stronger reference models significantly improve token selection accuracy and overall optimization effectiveness. Comprehensive experiments on benchmarks such as Arena-Hard and MT-Bench validate the superiority of our Selective-DPO method over standard DPO and distillation-based baselines. Our findings highlight the importance of token-level optimization and reference model selection in advancing preference alignment for LLMs. The code is available at https://github.com/Dongzhijin/SDPO.