Auditing the Audit: Five Failure Modes in Benchmark-Validity Audits
For AI auditors and regulators relying on benchmark-based evaluations, the paper reveals that current audit practices are fragile and can produce misleading evidence, highlighting a need for more rigorous protocols.
The paper identifies five failure modes in perturbation-based construct-validity audits that can silently invalidate conclusions, and demonstrates via a case study on safety benchmarks that all tested audits fail a proposed due-diligence gate, with no cell reaching confirmatory status.
Governance frameworks ask AI providers and auditors for documented evaluation evidence, and perturbation-based construct-validity audits are a common form of that evidence. We argue the audits are themselves fragile: their conclusions can be silently manufactured by implementation details that readers cannot see in the reported numbers. We name five classes of pipeline failure and demonstrate each in a self-audit over safety benchmarks and open-weight instruction-tuned models. Under a unified six-point due-diligence gate, every cell lands in a non-confirmatory bucket, and no cell reaches confirmatory. The evidence here is a single two-model, five-benchmark case study, and F1--F5 is an illustrative, deliberately non-exhaustive starting taxonomy -- not a comprehensive partition of audit failures. We position the gate as a withholding and disclosure protocol for assurance-grade evidence, supplementary to (not a replacement for) classical construct-validity evidence, and not as a route to benchmark-validity verdicts.