Multimodal neural networks better explain multivoxel patterns in the hippocampus
This work addresses how neural networks can model brain activity, but it is incremental as it builds on existing models like CLIP.
The study tackled the problem of explaining fMRI activity in the human hippocampus by comparing multimodal and unimodal models, finding that multimodality is a key component for better explanation of multivoxel patterns.
The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et at., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.