Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
This work addresses the underexplored role of data preprocessing in responsible AI development for researchers and practitioners in LLM alignment, though it is incremental as it benchmarks existing methods rather than proposing new ones.
The paper tackles the problem of noisy human feedback degrading LLM alignment by introducing PrefCleanBench, the first comprehensive benchmark evaluating 13 preference data cleaning methods, uncovering key factors for success in improving alignment performance across diverse datasets and models.
Human feedback plays a pivotal role in aligning large language models (LLMs) with human preferences. However, such feedback is often noisy or inconsistent, which can degrade the quality of reward models and hinder alignment. While various automated data cleaning methods have been proposed to mitigate this issue, a systematic evaluation of their effectiveness and generalizability remains lacking. To bridge this gap, we introduce the first comprehensive benchmark for evaluating 13 preference data cleaning methods in the context of LLM alignment. PrefCleanBench offers a standardized protocol to assess cleaning strategies in terms of alignment performance and generalizability across diverse datasets, model architectures, and optimization algorithms. By unifying disparate methods and rigorously comparing them, we uncover key factors that determine the success of data cleaning in alignment tasks. This benchmark lays the groundwork for principled and reproducible approaches to improving LLM alignment through better data quality-highlighting the crucial but underexplored role of data preprocessing in responsible AI development. We release modular implementations of all methods to catalyze further research: https://github.com/deeplearning-wisc/PrefCleanBench.