Multi-Step Knowledge Interaction Analysis via Rank-2 Subspace Disentanglement
This work provides a framework for systematic analysis of knowledge interactions in LLMs, addressing a key issue in assessing explanation grounding for researchers and practitioners, though it is incremental as it builds on prior single-step and rank-1 subspace methods.
The paper tackled the problem of understanding how Large Language Models (LLMs) interact external context knowledge and internal parametric knowledge during multi-step natural language explanation generation, by proposing a rank-2 projection subspace for disentanglement. The result showed that diverse knowledge interactions are poorly captured in prior rank-1 methods but effectively represented in their formulation, with experiments on four QA datasets and three LLMs revealing that hallucinated explanations align with parametric knowledge, faithful ones balance both, and Chain-of-Thought prompting reduces parametric reliance.
Natural Language Explanations (NLEs) describe how Large Language Models (LLMs) make decisions, drawing on both external Context Knowledge (CK) and Parametric Knowledge (PK) stored in model weights. Understanding their interaction is key to assessing the grounding of NLEs, yet it remains underexplored. Prior work has largely examined only single-step generation, typically the final answer, and has modelled PK and CK interaction only as a binary choice in a rank-1 subspace. This overlooks richer forms of interaction, such as complementary or supportive knowledge. We propose a novel rank-2 projection subspace that disentangles PK and CK contributions more accurately and use it for the first multi-step analysis of knowledge interactions across longer NLE sequences. Experiments on four QA datasets and three open-weight instruction-tuned LLMs show that diverse knowledge interactions are poorly represented in a rank-1 subspace but are effectively captured in our rank-2 formulation. Our multi-step analysis reveals that hallucinated NLEs align strongly with the PK direction, context-faithful ones balance PK and CK, and Chain-of-Thought prompting for NLEs shifts generated NLEs toward CK by reducing PK reliance. This work provides the first framework for systematic studies of multi-step knowledge interactions in LLMs through a richer rank-2 subspace disentanglement. Code and data: https://github.com/copenlu/pk-ck-knowledge-disentanglement.