Accuracy, Stability, and Repeated-Run Reliability of Large Language Models on Deterministic Programming Tasks
For practitioners deploying LLMs on deterministic tasks, the paper reveals that standard accuracy metrics can be misleading and that repeated-run stability is critical for reliable evaluation.
The paper shows that run-level pass rate overstates retry-free coverage by up to 17.8 percentage points on deterministic programming tasks, and that this gap reverses model rankings among closely matched systems. The authors propose a repeated-run evaluation protocol and demonstrate that stability analysis is a necessary complement to accuracy reporting.
Run-level pass rate overstates retry-free coverage by up to 17.8 percentage points -- and the gap is largest precisely for mid-performing systems. We investigate this accuracy--stability relationship in large language model (LLM) evaluation for deterministic text-conditioned generation, using programming tasks as a concrete testbed. Standard code-generation benchmarks emphasize single-run accuracy or eventual success under repeated sampling, but many deployment settings also require stability: consistent outcomes across repeated invocations under the same task description. We present a repeated-run evaluation protocol with metrics for run-level accuracy, retry-free coverage, and per-problem variability. On a recency-based benchmark of 100 LeetCode-style problems, we evaluate 16 models from five provider families under two prompt templates with five repeated runs per problem, yielding 16,000 evaluation instances. Although run-level pass rate and perfect stability rate are strongly correlated (r=0.985), pass rate consistently exceeds retry-free coverage -- a gap that reaches 17.8 percentage points and reverses model rankings even among closely matched systems. Prompt effects are model-dependent rather than uniformly beneficial. These results suggest that repeated-run stability analysis is a necessary complement to conventional accuracy reporting for deterministic text-conditioned generation tasks.