A Statistical Framework for Alignment with Biased AI Feedback
This addresses the challenge of reducing reliance on expensive human feedback for AI alignment, offering practical solutions for large-scale deployment, though it is incremental by building on existing methods like DPO.
The paper tackles the problem of systematic bias in AI-generated preference labels used for aligning large language models, proposing two debiased alignment methods that substantially improve alignment efficiency and recover performance close to an oracle trained on fully human-labeled data in tasks like sentiment generation, summarization, and dialogue.
Modern alignment pipelines are increasingly replacing expensive human preference labels with evaluations from large language models (LLM-as-Judge). However, AI labels can be systematically biased compared to high-quality human feedback datasets. In this paper, we develop two debiased alignment methods within a general framework that accommodates heterogeneous prompt-response distributions and external human feedback sources. Debiased Direct Preference Optimization (DDPO) augments standard DPO with a residual-based correction and density-ratio reweighting to mitigate systematic bias, while retaining DPO's computational efficiency. Debiased Identity Preference Optimization (DIPO) directly estimates human preference probabilities without imposing a parametric reward model. We provide theoretical guarantees for both methods: DDPO offers a practical and computationally efficient solution for large-scale alignment, whereas DIPO serves as a robust, statistically optimal alternative that attains the semiparametric efficiency bound. Empirical studies on sentiment generation, summarization, and single-turn dialogue demonstrate that the proposed methods substantially improve alignment efficiency and recover performance close to that of an oracle trained on fully human-labeled data.