Hidden Failures in Robustness: Why Supervised Uncertainty Quantification Needs Better Evaluation
For researchers building uncertainty estimation probes for LLMs, the paper reveals that current methods are unreliable under distribution shift and provides guidance for more robust designs.
The paper evaluates supervised uncertainty probes for LLMs, finding poor robustness under distribution shift, especially for long-form generations. Middle-layer representations and token aggregation improve robustness, while architecture matters less.
Recent work has shown that the hidden states of large language models contain signals useful for uncertainty estimation and hallucination detection, motivating a growing interest in efficient probe-based approaches. Yet it remains unclear how robust existing methods are, and which probe designs provide uncertainty estimates that are reliable under distribution shift. We present a systematic study of supervised uncertainty probes across models, tasks, and OOD settings, training over 2,000 probes while varying the representation layer, feature type, and token aggregation strategy. Our evaluation highlights poor robustness in current methods, particularly in the case of long-form generations. We also find that probe robustness is driven less by architecture and more by the probe inputs. Middle-layer representations generalise more reliably than final-layer hidden states, and aggregating across response tokens is consistently more robust than relying on single-token features. These differences are often largely invisible in-distribution but become more important under distribution shift. Informed by our evaluation, we explore a simple hybrid back-off strategy for improving robustness, arguing that better evaluation is a prerequisite for building more robust probes.