Large Language Models Do NOT Really Know What They Don't Know
This work addresses a fundamental limitation in AI reliability for users relying on LLMs for factual information, showing it is incremental by building on prior claims about factuality signals.
The paper investigates whether large language models (LLMs) can internally distinguish factual from hallucinated outputs, finding that hallucinations tied to subject knowledge produce indistinguishable internal states from correct responses, while detached ones are detectable, revealing LLMs rely on recall patterns rather than truthfulness encoding.
Recent work suggests that large language models (LLMs) encode factuality signals in their internal representations, such as hidden states, attention weights, or token probabilities, implying that LLMs may "know what they don't know". However, LLMs can also produce factual errors by relying on shortcuts or spurious associations. These error are driven by the same training objective that encourage correct predictions, raising the question of whether internal computations can reliably distinguish between factual and hallucinated outputs. In this work, we conduct a mechanistic analysis of how LLMs internally process factual queries by comparing two types of hallucinations based on their reliance on subject information. We find that when hallucinations are associated with subject knowledge, LLMs employ the same internal recall process as for correct responses, leading to overlapping and indistinguishable hidden-state geometries. In contrast, hallucinations detached from subject knowledge produce distinct, clustered representations that make them detectable. These findings reveal a fundamental limitation: LLMs do not encode truthfulness in their internal states but only patterns of knowledge recall, demonstrating that "LLMs don't really know what they don't know".