Representation-based Broad Hallucination Detectors Fail to Generalize Out of Distribution
This work highlights a critical limitation in hallucination detection for AI safety, showing that existing methods are incremental and unreliable for real-world applications.
The paper found that current state-of-the-art hallucination detectors rely on spurious data correlations and fail to generalize out-of-distribution, performing no better than simple supervised methods and close to random in such cases.
We critically assess the efficacy of the current SOTA in hallucination detection and find that its performance on the RAGTruth dataset is largely driven by a spurious correlation with data. Controlling for this effect, state-of-the-art performs no better than supervised linear probes, while requiring extensive hyperparameter tuning across datasets. Out-of-distribution generalization is currently out of reach, with all of the analyzed methods performing close to random. We propose a set of guidelines for hallucination detection and its evaluation.