From Flat to Round: Redefining Brain Decoding with Surface-Based fMRI and Cortex Structure
This work improves brain decoding for neuroscience and computer vision applications, though it is incremental by building on existing methods with specific enhancements.
The paper tackled the problem of reconstructing visual stimuli from fMRI data by addressing the neglect of brain structure-function relationships and individual anatomical variations, resulting in superior reconstruction accuracy and interpretability compared to state-of-the-art methods.
Reconstructing visual stimuli from human brain activity (e.g., fMRI) bridges neuroscience and computer vision by decoding neural representations. However, existing methods often overlook critical brain structure-function relationships, flattening spatial information and neglecting individual anatomical variations. To address these issues, we propose (1) a novel sphere tokenizer that explicitly models fMRI signals as spatially coherent 2D spherical data on the cortical surface; (2) integration of structural MRI (sMRI) data, enabling personalized encoding of individual anatomical variations; and (3) a positive-sample mixup strategy for efficiently leveraging multiple fMRI scans associated with the same visual stimulus. Collectively, these innovations enhance reconstruction accuracy, biological interpretability, and generalizability across individuals. Experiments demonstrate superior reconstruction performance compared to SOTA methods, highlighting the effectiveness and interpretability of our biologically informed approach.