Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors
For researchers and developers of LLMs, this work provides a diagnostic framework to distinguish knowledge gaps from inference misalignment, enabling targeted mitigation strategies.
The paper investigates whether LLM hallucinations stem from missing knowledge or biased inference, showing that pretraining frequency imbalances can cause models to favor statistically salient but incorrect answers over prompt-constrained ones. Using TrapQA, they demonstrate that hallucinations can arise from biased latent inference rather than absent knowledge alone.
Large language models often produce hallucinated answers that violate prompt-level constraints. A key diagnostic question is whether these failures reflect missing knowledge, or whether the model has the relevant information but follows the wrong inference path. We study this phenomenon as inference misalignment: a mismatch between the answer supported by the prompt and the answer favored by statistically salient latent associations. We formalize this view with a latent key-task model, in which pretraining-frequency imbalance can cause a shortcut path to dominate the constraint-sensitive path and induce positive inference loss. The framework predicts two failure modes: task-retrieval bias in entity disambiguation and key-selection bias in action choice. We introduce TrapQA, a controlled diagnostic testbed with two components. ScientistQA tests disambiguation among similar scientists with supplementary factual probes, while Real-Life Constrained QA tests everyday constraint following under salient shortcuts. Our results show that hallucination can arise from biased latent inference rather than absent knowledge alone.