Misalignment Between Backpropagation and the Hierarchy of Brain Responses to Images
For neuroscientists and AI researchers, this work challenges the biological plausibility of backpropagation as a learning mechanism in the brain, though the findings are incremental as they confirm known representational alignment but highlight mechanistic differences.
The study tested whether backpropagated gradients from deep neural networks align with human brain responses to images, finding that while gradients can predict fMRI and MEG signals in higher-level visual cortex, their spatial and temporal organization diverges from brain hierarchies, suggesting different learning mechanisms.
Backpropagation is the core learning mechanism underlying deep learning. However, whether and how this algorithm is implemented in the brain remains highly debated. In particular, while forward activations of pretrained models reliably map onto the cortical hierarchy of visual processing, it is unknown whether backpropagated gradients exhibit a similar correspondence. Here, we address this question using functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) recordings of human brain responses to natural images. For this, we extend standard encoding analyses of forward activations to map backpropagated gradients onto neural data. Focusing on a recent self-supervised vision model (DINOv3) and reproducing results on eight vision models, we find that backpropagated gradients can reliably predict both fMRI and MEG signals, specifically in higher-level visual cortex and for later latencies. However, the spatial and temporal organization of these backpropagated gradients in the brain diverges from the patterns expected under a biologically plausible backpropagation mechanism: specifically, both the order in which gradients are computed and their spatial organization diverge from the temporal and spatial hierarchies of the human brain. Together, these results suggest that, although deep networks and the brain may share similar representational content, they likely rely on fundamentally different mechanisms to learn those representations.