Do Repetitions Matter? Strengthening Reliability in LLM Evaluations
This addresses the problem of unreliable model comparisons for practitioners in AI, though it is incremental as it builds on existing evaluation practices.
The paper tackled the problem of unreliable LLM evaluations due to single stochastic runs by re-evaluating eight models on the AI4Math Benchmark with three independent runs. The result showed that single-run leaderboards are brittle, with 83% of slices inverting pairwise ranks, and that two runs remove about 83% of these inversions.
LLM leaderboards often rely on single stochastic runs, but how many repetitions are required for reliable conclusions remains unclear. We re-evaluate eight state-of-the-art models on the AI4Math Benchmark with three independent runs per setting. Using mixed-effects logistic regression, domain-level marginal means, rank-instability analysis, and run-to-run reliability, we assessed the value of additional repetitions. Our findings shows that Single-run leaderboards are brittle: 10/12 slices (83\%) invert at least one pairwise rank relative to the three-run majority, despite a zero sign-flip rate for pairwise significance and moderate overall interclass correlation. Averaging runs yields modest SE shrinkage ($\sim$5\% from one to three) but large ranking gains; two runs remove $\sim$83\% of single-run inversions. We provide cost-aware guidance for practitioners: treat evaluation as an experiment, report uncertainty, and use $\geq 2$ repetitions under stochastic decoding. These practices improve robustness while remaining feasible for small teams and help align model comparisons with real-world reliability.