Cross-Layer Attention Probing for Fine-Grained Hallucination Detection
This addresses reliability concerns for LLM users by providing a method to detect and mitigate hallucinations, though it is incremental as it builds on existing activation probing techniques.
The paper tackles the problem of detecting hallucinations in Large Language Models (LLMs) by proposing Cross-Layer Attention Probing (CLAP), which improves detection accuracy across multiple models and tasks, enabling fine-grained disambiguation between hallucinated and non-hallucinated responses.
With the large-scale adoption of Large Language Models (LLMs) in various applications, there is a growing reliability concern due to their tendency to generate inaccurate text, i.e. hallucinations. In this work, we propose Cross-Layer Attention Probing (CLAP), a novel activation probing technique for hallucination detection, which processes the LLM activations across the entire residual stream as a joint sequence. Our empirical evaluations using five LLMs and three tasks show that CLAP improves hallucination detection compared to baselines on both greedy decoded responses as well as responses sampled at higher temperatures, thus enabling fine-grained detection, i.e. the ability to disambiguate hallucinations and non-hallucinations among different sampled responses to a given prompt. This allows us to propose a detect-then-mitigate strategy using CLAP to reduce hallucinations and improve LLM reliability compared to direct mitigation approaches. Finally, we show that CLAP maintains high reliability even when applied out-of-distribution.