What Single-Prompt Accuracy Misses: A Multi-Variant Reliability Audit of Language Models
For practitioners and researchers evaluating language model reliability, this paper demonstrates that standard single-prompt accuracy metrics can miss systematic failures, urging explicit reporting of calibration definitions, evaluator logic, and prompt robustness.
This paper audits 10 instruct language models across 5 benchmarks and 5 prompt variants, finding that evaluation design (e.g., calibration definition, evaluator logic) can materially change conclusions: switching ECE token definition changes calibration by mean absolute 0.149, and a chain-of-thought prompt with a first-character evaluator reduces ARC-Challenge accuracy by 72-88% (recoverable via repair). Confidence signals are fragile, and prompt robustness does not correlate with model size.
Single-prompt accuracy is the dominant way to benchmark language models, but it can miss reliability failures that matter. We evaluate a 15-model open-weight corpus, with the main reliability analyses focused on 10 instruct models across five classification and reasoning benchmarks under five prompt variants each, measuring accuracy, token-probability calibration, verbal-confidence calibration, verbal parse rate, and prompt-perturbation spread for every (model x dataset x variant) cell. We find three broad results. First, evaluation design can materially change the conclusion. Switching Expected Calibration Error (ECE) token from a raw to a label-set-normalised definition changes per-cell calibration by a mean absolute 0.149. More strikingly, pairing a chain-of-thought prompt with a first-character evaluator on ARC-Challenge reduces apparent accuracy by 72-88% across all five primary models; two independent repair procedures recover 93.8% and 102.7% of the lost performance, indicating an evaluator-side rather than model-side failure. Second, confidence signals are fragile. On MMLU-Pro, every primary model verbally reports confidence substantially above both its accuracy and its token-probability confidence on the same rows, and verbal parse rate can collapse for a single model on a single prompt variant. Third, prompt robustness does not track parameter count reliably. Across 10 instruct models, the correlation between model size and prompt-perturbation spread ranges from -0.244 to 0.474 across benchmarks. Taken together, these results show that reliability conclusions for small language models depend not only on the model being evaluated, but also on the evaluation pipeline used to measure it. We argue that calibration definitions, evaluator logic, verbal parseability, and prompt robustness should be reported explicitly when making reliability claims.