Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection
This addresses reliability issues in sensitive LLM applications by providing a more efficient hallucination detection method, though it builds incrementally on existing consistency-based approaches.
The paper tackles the problem of hallucination detection in black-box LLMs by showing that self-consistency methods have limited improvement potential and proposing a two-stage algorithm that uses cross-model consistency with a verifier LLM only for uncertain cases. This approach maintains high detection performance while reducing computational cost by 40% compared to always using the verifier.
Large Language Models (LLMs) suffer from hallucination problems, which hinder their reliability in sensitive applications. In the black-box setting, several self-consistency-based techniques have been proposed for hallucination detection. We empirically study these techniques and show that they achieve performance close to that of a supervised (still black-box) oracle, suggesting little room for improvement within this paradigm. To address this limitation, we explore cross-model consistency checking between the target model and an additional verifier LLM. With this extra information, we observe improved oracle performance compared to purely self-consistency-based methods. We then propose a budget-friendly, two-stage detection algorithm that calls the verifier model only for a subset of cases. It dynamically switches between self-consistency and cross-consistency based on an uncertainty interval of the self-consistency classifier. We provide a geometric interpretation of consistency-based hallucination detection methods through the lens of kernel mean embeddings, offering deeper theoretical insights. Extensive experiments show that this approach maintains high detection performance while significantly reducing computational cost.