Mechanistic Interpretability of Brain-to-Speech Models Across Speech Modes
This provides a causal explanation for how speech modality information is organized in brain-to-speech models, which is incremental for neuroscience and brain-computer interfaces.
The study investigated the internal mechanisms of brain-to-speech decoding models across vocalized, mimed, and imagined speech modes, finding that speech modes lie on a shared continuous causal manifold and cross-mode transfer is mediated by compact, layer-specific subspaces rather than diffuse activity.
Brain-to-speech decoding models demonstrate robust performance in vocalized, mimed, and imagined speech; yet, the fundamental mechanisms via which these models capture and transmit information across different speech modalities are less explored. In this work, we use mechanistic interpretability to causally investigate the internal representations of a neural speech decoder. We perform cross-mode activation patching of internal activations across speech modes, and use tri-modal interpolation to examine whether speech representations vary discretely or continuously. We use coarse-to-fine causal tracing and causal scrubbing to find localized causal structure, allowing us to find internal subspaces that are sufficient for cross-mode transfer. In order to determine how finely distributed these effects are within layers, we perform neuron-level activation patching. We discover that small but not distributed subsets of neurons, rather than isolated units, affect the cross-mode transfer. Our results show that speech modes lie on a shared continuous causal manifold, and cross-mode transfer is mediated by compact, layer-specific subspaces rather than diffuse activity. Together, our findings give a causal explanation for how speech modality information is organized and used in brain-to-speech decoding models, revealing hierarchical and direction-dependent representational structure across speech modes.