Deep driven fMRI decoding of visual categories
This work addresses the problem of fMRI decoding for visual categories, which is incremental as it combines existing fMRI and deep learning methods in a novel hybrid approach.
The paper tackled the challenge of decoding visual categories from fMRI data by linking fMRI representations with deep features from a CNN using Kernel Canonical Correlation Analysis, resulting in effective semantic visual category distinction based solely on brain imaging data.
Deep neural networks have been developed drawing inspiration from the brain visual pathway, implementing an end-to-end approach: from image data to video object classes. However building an fMRI decoder with the typical structure of Convolutional Neural Network (CNN), i.e. learning multiple level of representations, seems impractical due to lack of brain data. As a possible solution, this work presents the first hybrid fMRI and deep features decoding approach: collected fMRI and deep learnt representations of video object classes are linked together by means of Kernel Canonical Correlation Analysis. In decoding, this allows exploiting the discriminatory power of CNN by relating the fMRI representation to the last layer of CNN (fc7). We show the effectiveness of embedding fMRI data onto a subspace related to deep features in distinguishing semantic visual categories based solely on brain imaging data.