Alignment-Aware Decoding
This addresses the problem of model alignment for natural language processing practitioners, offering a practical solution in data-constrained settings, though it is incremental as it builds on existing preference optimization methods.
The paper tackles the challenge of aligning large language models by introducing alignment-aware decoding (AAD), a method that enhances alignment directly at inference without specialized training, and it consistently outperforms baselines across benchmarks and model scales.
Alignment of large language models remains a central challenge in natural language processing. Preference optimization has emerged as a popular and effective method for improving alignment, typically through training-time or prompt-based interventions. In this paper, we introduce alignment-aware decoding (AAD), a method to enhance model alignment directly at inference. Theoretically, AAD can be interpreted as implicit reward optimization, yet it requires no specialized training beyond the standard DPO setup. Empirically, AAD consistently outperforms strong baselines across diverse alignment benchmarks and model scales. Moreover, in data-constrained settings, AAD can produce high-quality synthetic data to improve alignment under standard decoding, providing a practical solution when labeled data is limited.