Mitigating the Bias of Large Language Model Evaluation
This addresses bias in LLM evaluation for researchers and practitioners, but it is incremental as it builds on existing bias benchmarks and methods.
The paper tackled the problem of bias in LLM-as-a-Judge evaluations, where judges favor superficial quality over instruction following, and proposed calibration and contrastive training methods that mitigated the bias by a large margin while maintaining satisfactory accuracy.
Recently, there has been a trend of evaluating the Large Language Model (LLM) quality in the flavor of LLM-as-a-Judge, namely leveraging another LLM to evaluate the current output quality. However, existing judges are proven to be biased, namely they would favor answers which present better superficial quality (such as verbosity, fluency) while ignoring the instruction following ability. In this work, we propose systematic research about the bias of LLM-as-a-Judge. Specifically, for closed-source judge models, we apply calibration to mitigate the significance of superficial quality, both on probability level and prompt level. For open-source judge models, we propose to mitigate the bias by contrastive training, with curated negative samples that deviate from instruction but present better superficial quality. We apply our methods on the bias evaluation benchmark, and experiment results show our methods mitigate the bias by a large margin while maintaining a satisfactory evaluation accuracy.