wDPO: Winsorized Direct Preference Optimization for Robust LLM Alignment
This work provides an incremental improvement in robust LLM alignment for practitioners dealing with noisy preference datasets, offering a more nuanced approach to handling different noise types.
This paper introduces wDPO, a robust LLM alignment method that addresses noisy preference data by using a hierarchical winsorization strategy. It identifies heterogeneous noise patterns using implicit margins and applies data-level corrections for hard noise and gradient-level winsorization for ambiguous comparisons. Experiments on PKU-SafeRLHF and safety benchmarks show wDPO consistently improves alignment quality and robustness over DPO and its variants, especially under label-flip noise.
Direct Preference Optimization (DPO) aligns large language models by optimizing pairwise preferences and has shown remarkable effectiveness as a simple and scalable alternative to RLHF. However, in practice, preference data are often noisy. Existing robust variants of DPO mainly rely on uniform objective modifications or global reweighting. While partially effective, these methods treat noisy samples as a homogeneous source of uncertainty and fail to distinguish between different noise types, leading to sub-optimal alignment robustness. In this work, we show that robust preference alignment benefits from addressing different noise types with targeted interventions rather than uniform regularization. We propose winsorized Direct Preference Optimization~(wDPO), a robust LLM alignment approach with hierarchical winsorization. Specifically, wDPO adopts a reward-free hierarchical intervention strategy that leverages only signals already available during DPO training. It first uses the implicit margin from DPO log-ratio to identify heterogeneous noise patterns without relying on external reward models. For hard noise, wDPO performs a data-level intervention by sparsely correcting strongly inconsistent preference pairs. For ambiguous comparisons, it applies a gradient-level intervention through soft winsorization, capping extreme losses in the high-loss tail to prevent weakly informative samples from dominating gradient updates. Extensive experiments on PKU-SafeRLHF and multiple external safety benchmarks demonstrate that wDPO consistently improves preference alignment quality and robustness over vanilla DPO and strong DPO-family baselines, with particularly pronounced gains under controlled label-flip noise.