Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregation
For multi-stakeholder alignment tasks, this work identifies and mitigates a previously overlooked source of instability in LLM-based preference aggregation.
The paper shows that holistic LLM judges conflate utility estimation and aggregation, causing unstable weighting noise that grows with stakeholder count. The proposed DecompR method fixes weights before scoring and estimates utilities independently, reducing noise.
Multi-stakeholder tasks require one output to satisfy users with conflicting preferences. Holistic LLM judges conflate utility estimation and utility aggregation, yielding unstable implicit weights. We show empirically and theoretically that this aggregation-specific \emph{weighting noise} can create large score shifts when stakeholder satisfaction is dispersed; in our experiments, these weight-induced shifts also increase with stakeholder count. We propose \textsc{DecompR}: counterfactual-calibrated weights are fixed from query structure before candidate scoring, while per-role utilities are estimated independently, removing candidate-dependent weight drift and reducing estimation noise.