Tail-Shape Estimation in LLM Evaluation Is Fragile: A Protocol for Diagnosing False Positives
For researchers using extreme-value theory in LLM evaluation, this work provides a diagnostic protocol to avoid false positive claims, revealing fragility in tail-shape estimation that the recent literature overlooks.
The paper introduces a pre-registered protocol for diagnosing false positives in tail-shape estimation for LLM evaluation, demonstrating that three distinct modes of false positives arise in a standard toxicity-evaluation setup, and rejects the headline tail-shape claim on both scorers tested.
Recent work motivates moving large language model (LLM) evaluation from mean-based to tail-aware metrics, including conditional value-at-risk and tail-index estimates of reward-model error. We ask whether the canonical extreme-value-theory tail-index parameter, which isolates how heavy a tail is from how large the tail mass is, adds discriminative information beyond the mean and a standard tail-magnitude statistic in LLM evaluation. We pre-register a protocol covering admissibility, goodness-of-fit, threshold-stability, and effect-size requirements for any positive tail-shape claim. The protocol is the contribution of this paper; the empirical study below is a demonstration of what its gates catch. Applied to a standard LLM toxicity-evaluation setup under two structurally different scorer families, the protocol catches three distinct modes of false positives that a naive analysis would have published, and rejects the headline tail-shape claim on both scorers. We conclude that tail-shape estimation in the LLM toxicity-evaluation setups we examined is more fragile than the recent literature suggests, and recommend the protocol as a starting point for tail-index claims in similar setups.