The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity
This addresses the problem of inconsistent LLM alignment due to preference heterogeneity for AI developers, though it appears incremental as it modifies an existing aggregation step.
The paper tackles the problem of LLM alignment being vulnerable to heterogeneity in human preferences by proposing the sign estimator, which replaces cross-entropy with binary classification loss. This method reduces angular estimation error by nearly 35% and decreases disagreement with true population preferences from 12% to 8% compared to standard RLHF in simulations.
Traditional LLM alignment methods are vulnerable to heterogeneity in human preferences. Fitting a naïve probabilistic model to pairwise comparison data (say over prompt-completion pairs) yields an inconsistent estimate of the population-average utility -a canonical measure of social welfare. We propose a new method, dubbed the sign estimator, that provides a simple, provably consistent, and efficient estimator by replacing cross-entropy with binary classification loss in the aggregation step. This simple modification recovers consistent ordinal alignment under mild assumptions and achieves the first polynomial finite-sample error bounds in this setting. In realistic simulations of LLM alignment using digital twins, the sign estimator substantially reduces preference distortion over a panel of simulated personas, cutting (angular) estimation error by nearly 35% and decreasing disagreement with true population preferences from 12% to 8% compared to standard RLHF. Our method also compares favorably to panel data heuristics that explicitly model user heterogeneity and require tracking individual-level preference data-all while maintaining the implementation simplicity of existing LLM alignment pipelines.