Approximating Human Preferences Using a Multi-Judge Learned System
This work addresses the problem of unreliable LLM judges for applications like RLHF and model routing, but it appears incremental as it builds on existing aggregation methods without claiming major breakthroughs.
The paper tackles the challenge of aligning LLM-based judges with human preferences by proposing a framework that aggregates outputs from multiple rubric-conditioned judges to model diverse, persona-based preferences, and it investigates performance against naive baselines and robustness through case studies on biases.
Aligning LLM-based judges with human preferences is a significant challenge, as they are difficult to calibrate and often suffer from rubric sensitivity, bias, and instability. Overcoming this challenge advances key applications, such as creating reliable reward models for Reinforcement Learning from Human Feedback (RLHF) and building effective routing systems that select the best-suited model for a given user query. In this work, we propose a framework for modeling diverse, persona-based preferences by learning to aggregate outputs from multiple rubric-conditioned judges. We investigate the performance of this approach against naive baselines and assess its robustness through case studies on both human and LLM-judges biases. Our primary contributions include a persona-based method for synthesizing preference labels at scale and two distinct implementations of our aggregator: Generalized Additive Model (GAM) and a Multi-Layer Perceptron (MLP).