Sparse Autoencoders Bridge The Deep Learning Model and The Brain
This work addresses the challenge of interpreting deep learning models by bridging them with neuroscience, offering insights into model-brain alignment, though it is incremental in applying existing methods to this domain.
The authors tackled the problem of aligning deep learning model representations with human brain fMRI responses by introducing SAE-BrainMap, a framework using sparse autoencoders to map model activations to cortical signals, achieving a maximum similarity of 0.76 and revealing hierarchical correspondences between model layers and the visual pathway.
We present SAE-BrainMap, a novel framework that directly aligns deep learning visual model representations with voxel-level fMRI responses using sparse autoencoders (SAEs). First, we train layer-wise SAEs on model activations and compute the correlations between SAE unit activations and cortical fMRI signals elicited by the same natural image stimuli with cosine similarity, revealing strong activation correspondence (maximum similarity up to 0.76). Depending on this alignment, we construct a voxel dictionary by optimally assigning the most similar SAE feature to each voxel, demonstrating that SAE units preserve the functional structure of predefined regions of interest (ROIs) and exhibit ROI-consistent selectivity. Finally, we establish fine-grained hierarchical mapping between model layers and the human ventral visual pathway, also by projecting voxel dictionary activations onto individual cortical surfaces, we visualize the dynamic transformation of the visual information in deep learning models. It is found that ViT-B/16$_{CLIP}$ tends to utilize low-level information to generate high-level semantic information in the early layers and reconstructs the low-dimension information later. Our results establish a direct, downstream-task-free bridge between deep neural networks and human visual cortex, offering new insights into model interpretability.