LLM Knowledge is Brittle: Truthfulness Representations Rely on Superficial Resemblance
This work reveals a fundamental limitation in how LLMs encode knowledge, showing their truthfulness representations rely on superficial resemblance rather than robust understanding, which is critical for improving reliability in real-world applications.
The researchers investigated whether the brittleness of Large Language Models (LLMs) stems from unstable internal knowledge representations by testing how truthfulness separability in representations degrades with superficial input transformations like typos or reformulations. They found that LLMs' ability to distinguish true from false statements collapses as samples become less similar to pre-training data, with separability dropping by up to 40% on out-of-distribution samples across four model families and five datasets.
For Large Language Models (LLMs) to be reliable, they must learn robust knowledge that can be generally applied in diverse settings -- often unlike those seen during training. Yet, extensive research has shown that LLM performance can be brittle, with models exhibiting excessive sensitivity to trivial input variations. In this work, we explore whether this brittleness is a direct result of unstable internal knowledge representations. To explore this question, we build on previous work showing that LLM representations encode statement truthfulness -- i.e., true, factual statements can be easily separated from false, inaccurate ones. Specifically, we test the robustness of learned knowledge by evaluating representation separability on samples that have undergone superficial transformations to drive them out-of-distribution (OOD), such as typos or reformulations. By applying semantically-preserving perturbations, we study how separability degrades as statements become more OOD, across four LLM families, five evaluation datasets, and three knowledge probing methods. Our results reveal that internal representations of statement truthfulness collapse as the samples' presentations become less similar to those seen during pre-training. While LLMs can often distinguish between true and false statements when they closely resemble the pre-training data, this ability is highly dependent on the statement's exact surface form. These findings offer a possible explanation for brittle benchmark performance: LLMs may learn shallow, non-robust knowledge representations that allow for only limited generalizability. Our work presents a fundamental challenge for the utility of truthfulness probes, and more broadly, calls for further research on improving the robustness of learned knowledge representations.