When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
This work significantly reduces the cost of preference alignment for LLM developers and researchers by demonstrating that weak LLMs can effectively replace or augment human annotation efforts.
This paper explores using a weak LLM as an annotator for preference alignment, finding that selecting only highly confident samples from the weak LLM leads to better performance than full human annotations. They propose Confidence-Weighted Preference Optimization (CW-PO), which, with only 20% of human annotations, outperforms models trained with 100% human annotations under standard DPO.
Preference alignment is an essential step in adapting large language models (LLMs) to human values, but existing approaches typically depend on costly human annotations or large-scale API-based models. We explore whether a weak LLM can instead act as an effective annotator. We surprisingly find that selecting only a subset of a weak LLM's highly confident samples leads to substantially better performance than using full human annotations. Building on this insight, we propose Confidence-Weighted Preference Optimization (CW-PO), a general framework that re-weights training samples by a weak LLM's confidence and can be applied across different preference optimization objectives. Notably, the model aligned by CW-PO with just 20% of human annotations outperforms the model trained with 100% of annotations under standard DPO. These results suggest that weak LLMs, when paired with confidence weighting, can dramatically reduce the cost of preference alignment while even outperforming methods trained on fully human-labeled data.