Pre-Generation Hallucination Detection in Large Language Models via Soft-Target Attention Probing
For developers of large language models, this provides a more principled and effective method to detect hallucination risk before generation, enabling cost-saving abstention or routing decisions.
This work formulates pre-generation hallucination detection as a risk-estimation problem and introduces soft-target supervision based on the empirical answer error rate over stochastic samples, which is proven to be the unique unbiased minimum-variance estimator of per-prompt error probability. Attention probing adapted to this setting outperforms linear probing on short-answer tasks across three benchmarks and five models, with soft-target supervision consistently improving detection quality.
Detecting hallucination risk before generation enables abstention, retrieval augmentation, and routing decisions without incurring the cost of decoding. While prior work has shown that such risk can be estimated from a model's internal representations, existing approaches treat this as binary classification over a single decoded output. We instead formulate it as a risk-estimation problem. Under this formulation, we introduce soft-target supervision based on the empirical answer error rate over stochastically sampled outputs - an estimator we prove to be the unique unbiased minimum-variance estimator of the model's per-prompt error probability under its sampling distribution. We further adapt attention probing to the pre-generation setting, enabling the detector to selectively aggregate hallucination-relevant prompt representations. Across three question-answering benchmarks and five models, attention probing outperforms linear probing on short-answer tasks. Replacing binary labels with soft-target supervision further and consistently improves detection quality.