Causality $\neq$ Invariance: Function and Concept Vectors in LLMs
This work addresses the understanding of concept representation in LLMs for researchers, revealing a distinction between task-driven and abstract representations, though it is incremental in refining existing methods.
The paper tackled the problem of whether large language models (LLMs) represent concepts abstractly by revisiting Function Vectors (FVs) and identifying Concept Vectors (CVs) as more stable representations, showing that FVs are not invariant across input formats while CVs generalize better out-of-distribution.
Do large language models (LLMs) represent concepts abstractly, i.e., independent of input format? We revisit Function Vectors (FVs), compact representations of in-context learning (ICL) tasks that causally drive task performance. Across multiple LLMs, we show that FVs are not fully invariant: FVs are nearly orthogonal when extracted from different input formats (e.g., open-ended vs. multiple-choice), even if both target the same concept. We identify Concept Vectors (CVs), which carry more stable concept representations. Like FVs, CVs are composed of attention head outputs; however, unlike FVs, the constituent heads are selected using Representational Similarity Analysis (RSA) based on whether they encode concepts consistently across input formats. While these heads emerge in similar layers to FV-related heads, the two sets are largely distinct, suggesting different underlying mechanisms. Steering experiments reveal that FVs excel in-distribution, when extraction and application formats match (e.g., both open-ended in English), while CVs generalize better out-of-distribution across both question types (open-ended vs. multiple-choice) and languages. Our results show that LLMs do contain abstract concept representations, but these differ from those that drive ICL performance.