How Sensitive Are Safety Benchmarks to Judge Configuration Choices?
For researchers and practitioners using LLM-based safety benchmarks, this work reveals that judge prompt configuration is a substantial source of measurement variance that must be accounted for.
This paper shows that the choice of judge prompt wording in safety benchmarks like HarmBench can shift measured harmful-response rates by up to 24.2 percentage points, even when the judge model is fixed, and that model safety rankings are moderately unstable (mean Kendall tau = 0.89).
Safety benchmarks such as HarmBench rely on LLM judges to classify model responses as harmful or safe, yet the judge configuration, namely the combination of judge model and judge prompt, is typically treated as a fixed implementation detail. We show this assumption is problematic. Using a 2 x 2 x 3 factorial design, we construct 12 judge prompt variants along two axes, evaluation structure and instruction framing, and apply them using a single judge model, Claude Sonnet 4-6, producing 28,812 judgments over six target models and 400 HarmBench behaviors. We find that prompt wording alone, holding the judge model fixed, shifts measured harmful-response rates by up to 24.2 percentage points, with even within-condition surface rewording causing swings of up to 20.1 percentage points. Model safety rankings are moderately unstable, with mean Kendall tau = 0.89, and category-level sensitivity ranges from 39.6 percentage points for copyright to 0 percentage points for harassment. A supplementary multi-judge experiment using three judge models shows that judge-model choice adds further variance. Our results demonstrate that judge prompt wording is a substantial, previously under-examined source of measurement variance in safety benchmarking.