Sparse components distinguish visual pathways & their alignment to neural networks
This work addresses the problem of understanding neural tuning alignment in visual systems for neuroscience and AI researchers, offering a novel method for more sensitive measurement, though it is incremental in improving alignment analysis.
The study tackled the inconsistency between deep neural networks modeling the entire visual system well and known functional differences across visual pathways by applying a sparse decomposition approach, finding distinct component response profiles across ventral, dorsal, and lateral streams and introducing Sparse Component Alignment (SCA) to measure alignment, revealing that standard DNNs are more aligned with ventral than dorsal or lateral representations.
The ventral, dorsal, and lateral streams in high-level human visual cortex are implicated in distinct functional processes. Yet, deep neural networks (DNNs) trained on a single task model the entire visual system surprisingly well, hinting at common computational principles across these pathways. To explore this inconsistency, we applied a novel sparse decomposition approach to identify the dominant components of visual representations within each stream. Consistent with traditional neuroscience research, we find a clear difference in component response profiles across the three visual streams -- identifying components selective for faces, places, bodies, text, and food in the ventral stream; social interactions, implied motion, and hand actions in the lateral stream; and some less interpretable components in the dorsal stream. Building on this, we introduce Sparse Component Alignment (SCA), a new method for measuring representational alignment between brains and machines that better captures the latent neural tuning of these two visual systems. Using SCA, we find that standard visual DNNs are more aligned with the ventral than either dorsal or lateral representations. SCA reveals these distinctions with greater resolution than conventional population-level geometry, offering a measure of representational alignment that is sensitive to a system's underlying axes of neural tuning.