Quantifying Ranking Uncertainty in LLM Benchmarks
Provides a method to quantify ranking uncertainty in LLM benchmarks, which is important for practitioners who rely on leaderboards to select models.
The paper analyzes sources of uncertainty in the MMLU benchmark and modifies hypothesis tests to account for them, showing that ranking variability across subjects is substantial and should be considered when comparing LLMs.
Pretrained models are typically ranked on multi-task leaderboards to assess their effectiveness across diverse tasks. Rank confidence intervals were recently introduced as a method to quantify the uncertainty in these rankings by aggregating pairwise hypothesis tests. In this work, we analyze the sources of uncertainty in the knowledge evaluation benchmark MMLU and show how hypothesis tests can be modified to account for their effects. We demonstrate that ranking variability across MMLU subjects is substantial and should be considered when comparing LLMs or identifying the top-performing models.