Brain-aligning of semantic vectors improves neural decoding of visual stimuli
This work addresses the challenge of accurately decoding neural information in neuroscience, offering a method to enhance brain decoding algorithms, though it is incremental as it builds on existing representation learning.
The authors tackled the problem of brain decoding by fine-tuning pretrained semantic vectors to align with neural representations, resulting in improved zero-shot decoding accuracy across fMRI, MEG, and ECoG datasets.
The development of algorithms to accurately decode of neural information is a long-standing effort in the field of neuroscience. Brain decoding is typically employed by training machine learning models to map neural data onto a preestablished vector representation of stimulus features. These vectors are usually derived from image- and/or text-based feature spaces. Nonetheless, the intrinsic characteristics of these vectors might be fundamentally different than those encoded by the brain, limiting the ability of algorithms to accurately learn this mapping. To address this issue, here, we propose a representation learning framework, called brain-aligning of semantic vectors, that fine-tunes pretrained feature vectors to better align with the structure of neural representations of visual stimuli in the human brain. We trained this model with functional magnetic resonance imaging (fMRI) data representing 150 visual stimulus categories; then, we performed zero-shot brain decoding on 1) fMRI, 2) magnetoencephalography (MEG), and 3) electrocorticography (ECoG) data reflecting neural representations of visual stimuli. By using fMRI-based brain-aligned vectors, the zero-shot decoding accuracy all three neuroimaging datasets increased. This finding underscores the potential of leveraging a richer array of brainderived features to increase the performance of brain decoding algorithms.