Factual Retrieval in LLMs Is a Redundant, Distributed and Non-Contiguous Process
For researchers studying knowledge storage and editing in LLMs, this work reveals that factual retrieval is far more distributed and redundant than previously assumed, challenging the effectiveness of current localization-based editing approaches.
The paper investigates how LLMs compute factual attributes from entity representations, finding that the process is non-contiguous, redundant, and distributed across multiple functionally-equivalent paths. Using LLaMA 3.1 8B and Qwen3 8B, they show that attribute computation paths skip layers and have high redundancy, which may explain why localization-based editing methods often fail.
Large language models (LLMs) store and recall factual knowledge, yet the precise mechanism of how entity representations are transformed to enable specific attribute retrieval remains underexplored. In this work, we investigate this mechanism through the lens of an "attribute-computation path"-a sequence of computational steps over the entity representation required to elicit a target attribute. We then propose an iterative patching protocol to identify a minimal subset of layers necessary for this computation. Applying our method to LLaMA 3.1 8B and Qwen3 8B, we find that these paths are non-contiguous, often skipping layers, and that models possess multiple, functionally-equivalent paths for the same entity and fact, highlighting a high degree of redundancy in attribute computation. This implies that knowledge computation is highly distributed, potentially explaining the localization-editing mismatch and suggesting that knowledge storage and retrieval in LLMs is far from being well understood.