DecodeShare: Tracing the Shared Subspace of LLM Decode-Time Decisions
For LLM researchers and practitioners, this work reveals a task-general causal channel at decode time with practical implications for activation steering, though it is an incremental discovery of a known phenomenon (shared subspaces) in a new context.
DecodeShare identifies a low-dimensional subspace in LLM decode-time hidden states that is shared across tasks and causally important for decision performance, showing that disturbing it degrades performance more than disturbing prefill-derived or random subspaces. The subspace also affects activation steering, enabling more reliable signal at decode time.
Large language models (LLMs) handle many tasks with one set of parameters, but under KV-cached inference it is unclear what task-general structure, if any, is used at decode time rather than during prefill. We propose DecodeShare, a protocol that identifies a low-dimensional subspace consistently shared across tasks in decode-time hidden states, and then tests its causal role by removing that subspace only during decoding. In our experiments, disturbing the discovered shared subspace degrades decision performance far more than disturbing either a prefill-derived or random subspace under the same intervention budget. We further show this decode-shared subspace has practical consequences for activation steering: common steering directions can overlap the task-general decode channel. Projecting out this shared subspace directly separates the functional roles of the two components, while evaluating steering vectors at decode-time yields more reliable signal for downstream deployment than prefill-based proxies. Despite its compactness, the shared subspace can serve as a high-leverage causal channel at decode time. Code is available at: https://github.com/Zishan-Shao/decodeshare.git.