Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer
This addresses a critical safety issue for high-stakes domains like medicine or law, where model certainty is used as a reliability proxy, though it is incremental in focusing on a specific hallucination type.
The paper investigates a distinct type of hallucination in large language models called CHOKE, where models answer correctly but produce high-certainty hallucinations under trivial perturbations, and shows that existing mitigation methods perform worse on these examples, with a new probing-based method outperforming them.
Prior work on large language model (LLM) hallucinations has associated them with model uncertainty or inaccurate knowledge. In this work, we define and investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation, which can happen in real-world settings, causes it to produce a hallucinated response with high certainty. This phenomenon, which we dub CHOKE (Certain Hallucinations Overriding Known Evidence), is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability. We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations. This difference leads existing mitigation methods to perform worse on CHOKE examples than on general hallucinations. Finally, we introduce a probing-based mitigation that outperforms existing methods on CHOKE hallucinations. These findings reveal an overlooked aspect of hallucinations, emphasizing the need to understand their origins and improve mitigation strategies to enhance LLM safety. The code is available at https://github.com/technion-cs-nlp/Trust_me_Im_wrong .