Generic decoding of seen and imagined objects using hierarchical visual features
This work addresses the challenge of brain-based information retrieval for arbitrary objects, which is incremental as it builds on prior fMRI decoding methods.
The authors tackled the problem of decoding arbitrary seen and imagined objects from fMRI data, which was previously limited to training examples, by using hierarchical visual features from a convolutional neural network. They achieved this by predicting features from fMRI patterns and identifying object categories beyond the training set, with accuracy varying by visual area level.
Object recognition is a key function in both human and machine vision. While recent studies have achieved fMRI decoding of seen and imagined contents, the prediction is limited to training examples. We present a decoding approach for arbitrary objects, using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features including those from a convolutional neural network can be predicted from fMRI patterns and that greater accuracy is achieved for low/high-level features with lower/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, the decoding of imagined objects reveals progressive recruitment of higher to lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.