Measuring Human Preferences in RLHF is a Social Science Problem
For the ML community working on alignment, this paper highlights a critical overlooked problem—measurement validity in preference elicitation—and offers a framework to improve RLHF reliability.
The paper argues that RLHF's assumption that annotation responses reflect genuine human preferences is flawed, drawing on behavioral science to show that people often produce non-attitudes, constructed preferences, and measurement artifacts. It provides a taxonomy and diagnostic tools to distinguish valid preferences from noise, suggesting current RLHF may model artifacts as human values.
RLHF assumes that annotation responses reflect genuine human preferences. We argue this assumption warrants systematic examination, and that behavioral science offers frameworks that bring clarity to when it holds and when it breaks down. Behavioral scientists have documented for sixty years that people routinely produce responses without holding genuine opinions, construct preferences on the spot based on contextual cues, and interpret identical questions differently. These phenomena are pervasive for precisely the value-laden judgments that matter most for alignment, yet this literature has not yet been systematically integrated into ML practice. We argue that the ML community must treat measurement validity as logically prior to preference aggregation. Specifically, we contend that measuring human preferences in RLHF is a social science problem. We present a taxonomy distinguishing genuine preferences from non-attitudes, constructed preferences, and measurement artifacts, along with diagnostic approaches for detecting each. This framework has two important implications. First, it raises the question of whether current RLHF practice may be systematically modeling noise as signal and elicitation artifacts as human values. Second, it provides a path forward by suggesting diagnostic tools that can distinguish valid preferences from artifacts before they enter the training pipeline.