Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judge
This work addresses the reliability issue in automated evaluation for LLMs, which is crucial for researchers and practitioners in AI and NLP, though it is incremental as it builds on existing LLM-as-a-Judge paradigms.
The paper tackles the problem of single-point evaluations in LLM-as-a-Judge methods, which overlook diversity and uncertainty in human judgments, by proposing a training framework that aligns LLM-generated judgment distributions with human distributions, resulting in significant improvements in alignment quality, evaluation accuracy, and robustness across various tasks and backbones.
LLMs have emerged as powerful evaluators in the LLM-as-a-Judge paradigm, offering significant efficiency and flexibility compared to human judgments. However, previous methods primarily rely on single-point evaluations, overlooking the inherent diversity and uncertainty in human evaluations. This approach leads to information loss and decreases the reliability of evaluations. To address this limitation, we propose a novel training framework that explicitly aligns the LLM-generated judgment distribution with empirical human distributions. Specifically, we propose a distributional alignment objective based on KL divergence, combined with an auxiliary cross-entropy regularization to stabilize the training process. Furthermore, considering that empirical distributions may derive from limited human annotations, we incorporate adversarial training to enhance model robustness against distribution perturbations. Extensive experiments across various LLM backbones and evaluation tasks demonstrate that our framework significantly outperforms existing closed-source LLMs and conventional single-point alignment methods, with improved alignment quality, evaluation accuracy, and robustness.