Factual Self-Awareness in Language Models: Representation, Robustness, and Scaling
This addresses the problem of factual incorrectness in LLM-generated content for users relying on AI for accurate information, though it is incremental in building on prior fact-checking research.
The study investigated whether large language models (LLMs) have an internal mechanism to gauge factual correctness during generation, finding that they encode linear features in the Transformer's residual stream that predict accurate recall of entity-relation-attribute triplets, with robustness to formatting and scaling effects showing self-awareness peaks in intermediate layers.
Factual incorrectness in generated content is one of the primary concerns in ubiquitous deployment of large language models (LLMs). Prior findings suggest LLMs can (sometimes) detect factual incorrectness in their generated content (i.e., fact-checking post-generation). In this work, we provide evidence supporting the presence of LLMs' internal compass that dictate the correctness of factual recall at the time of generation. We demonstrate that for a given subject entity and a relation, LLMs internally encode linear features in the Transformer's residual stream that dictate whether it will be able to recall the correct attribute (that forms a valid entity-relation-attribute triplet). This self-awareness signal is robust to minor formatting variations. We investigate the effects of context perturbation via different example selection strategies. Scaling experiments across model sizes and training dynamics highlight that self-awareness emerges rapidly during training and peaks in intermediate layers. These findings uncover intrinsic self-monitoring capabilities within LLMs, contributing to their interpretability and reliability.