Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation
This addresses a trade-off in using reasoning for LLM reliability, but it is incremental as it builds on existing CoT and detection methods.
The paper tackles the problem of hallucinations in Large Language Models (LLMs) by evaluating how Chain-of-Thought (CoT) prompting affects hallucination detection, finding that while CoT reduces hallucination frequency, it obscures detection signals and impairs detection methods.
Large Language Models (LLMs) often exhibit \textit{hallucinations}, generating factually incorrect or semantically irrelevant content in response to prompts. Chain-of-Thought (CoT) prompting can mitigate hallucinations by encouraging step-by-step reasoning, but its impact on hallucination detection remains underexplored. To bridge this gap, we conduct a systematic empirical evaluation. We begin with a pilot experiment, revealing that CoT reasoning significantly affects the LLM's internal states and token probability distributions. Building on this, we evaluate the impact of various CoT prompting methods on mainstream hallucination detection methods across both instruction-tuned and reasoning-oriented LLMs. Specifically, we examine three key dimensions: changes in hallucination score distributions, variations in detection accuracy, and shifts in detection confidence. Our findings show that while CoT prompting helps reduce hallucination frequency, it also tends to obscure critical signals used for detection, impairing the effectiveness of various detection methods. Our study highlights an overlooked trade-off in the use of reasoning. Code is publicly available at: https://github.com/ECNU-Text-Computing/cot-hallu-detect .