Towards Interpretable Visual Decoding with Attention to Brain Representations
This work addresses the need for interpretable brain-to-image decoding for brain science researchers, offering an incremental improvement by bypassing intermediate feature spaces and enhancing transparency.
The authors tackled the problem of decoding visual stimuli from brain activity by proposing NeuroAdapter, a framework that directly conditions a latent diffusion model on brain representations, achieving competitive reconstruction quality on fMRI datasets while providing greater transparency into the generation process.
Recent work has demonstrated that complex visual stimuli can be decoded from human brain activity using deep generative models, helping brain science researchers interpret how the brain represents real-world scenes. However, most current approaches leverage mapping brain signals into intermediate image or text feature spaces before guiding the generative process, masking the effect of contributions from different brain areas on the final reconstruction output. In this work, we propose NeuroAdapter, a visual decoding framework that directly conditions a latent diffusion model on brain representations, bypassing the need for intermediate feature spaces. Our method demonstrates competitive visual reconstruction quality on public fMRI datasets compared to prior work, while providing greater transparency into how brain signals shape the generation process. To this end, we contribute an Image-Brain BI-directional interpretability framework (IBBI) which investigates cross-attention mechanisms across diffusion denoising steps to reveal how different cortical areas influence the unfolding generative trajectory. Our results highlight the potential of end-to-end brain-to-image decoding and establish a path toward interpreting diffusion models through the lens of visual neuroscience.