Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge
This addresses reliability issues in LLM-based evaluation for researchers and practitioners, though it is incremental as it builds on existing contrastive decoding methods.
The paper tackles the problem of score range bias in LLM-as-a-judge evaluations, where model outputs are sensitive to pre-defined score ranges, and shows that contrastive decoding mitigates this bias, achieving up to 11.3% relative improvement in Spearman correlation with human judgments.
Large Language Models (LLMs) are commonly used as evaluators in various applications, but the reliability of the outcomes remains a challenge. One such challenge is using LLMs-as-judges for direct assessment, i.e., assigning scores from a specified range without any references. We first show that this challenge stems from LLM judge outputs being associated with score range bias, i.e., LLM judge outputs are highly sensitive to pre-defined score ranges, preventing the search for optimal score ranges. We also show that similar biases exist among models from the same family. We then mitigate this bias through contrastive decoding, achieving up to 11.3% relative improvement on average in Spearman correlation with human judgments across different score ranges.