Decoding natural image stimuli from fMRI data with a surface-based convolutional network
This work addresses the challenge of decoding visual stimuli from noisy fMRI data, which could advance brain-computer interfaces and neuroscience research, though it appears incremental as it builds on existing methods like GANs.
The authors tackled the problem of reconstructing natural images from fMRI data by proposing Cortex2Image, a method that combines semantic feature decoding with fine-grained detail generation, achieving state-of-the-art semantic fidelity and good similarity to ground-truth stimuli.
Due to the low signal-to-noise ratio and limited resolution of functional MRI data, and the high complexity of natural images, reconstructing a visual stimulus from human brain fMRI measurements is a challenging task. In this work, we propose a novel approach for this task, which we call Cortex2Image, to decode visual stimuli with high semantic fidelity and rich fine-grained detail. In particular, we train a surface-based convolutional network model that maps from brain response to semantic image features first (Cortex2Semantic). We then combine this model with a high-quality image generator (Instance-Conditioned GAN) to train another mapping from brain response to fine-grained image features using a variational approach (Cortex2Detail). Image reconstructions obtained by our proposed method achieve state-of-the-art semantic fidelity, while yielding good fine-grained similarity with the ground-truth stimulus. Our code is available at: https://github.com/zijin-gu/meshconv-decoding.git.