SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models
This addresses the need for fairer and more reliable evaluation methods for large language models, though it is incremental as it builds on existing debiasing techniques.
The study tackled the problem of inflated scores in LLM evaluations due to selection biases in multiple-choice tasks by introducing SCOPE, a framework that estimates and mitigates position and label biases, resulting in consistent performance improvements and clearer confidence distributions across benchmarks.
Large Language Models (LLMs) can achieve inflated scores on multiple-choice tasks by exploiting inherent biases in option positions or labels, rather than demonstrating genuine understanding. This study introduces SCOPE, an evaluation framework designed to measure and mitigate such selection bias in a dataset-independent manner. By repeatedly invoking a null prompt that lacks semantic content, SCOPE estimates each model's unique position-bias distribution. It then redistributes the answer slot according to the inverse-bias distribution, thereby equalizing the lucky-rate, the probability of selecting the correct answer by chance. Furthermore, it prevents semantically similar distractors from being placed adjacent to the answer, thereby blocking near-miss guesses based on superficial proximity cues. Across multiple benchmark experiments, SCOPE consistently outperformed existing debiasing methods in terms of stable performance improvements and showed clearer confidence distributions over correct options. This framework thus offers a new standard for enhancing the fairness and reliability of LLM evaluations.