TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction
This work addresses the challenge of building a unified model of cognition across modalities and brain regions for neuroscience, though it appears incremental as it builds on existing foundational models.
The authors tackled the problem of predicting whole-brain fMRI responses to multimodal stimuli by introducing TRIBE, a deep neural network that combines text, audio, and video representations with a transformer, achieving first place in the Algonauts 2025 competition with a significant margin over competitors.
Historically, neuroscience has progressed by fragmenting into specialized domains, each focusing on isolated modalities, tasks, or brain regions. While fruitful, this approach hinders the development of a unified model of cognition. Here, we introduce TRIBE, the first deep neural network trained to predict brain responses to stimuli across multiple modalities, cortical areas and individuals. By combining the pretrained representations of text, audio and video foundational models and handling their time-evolving nature with a transformer, our model can precisely model the spatial and temporal fMRI responses to videos, achieving the first place in the Algonauts 2025 brain encoding competition with a significant margin over competitors. Ablations show that while unimodal models can reliably predict their corresponding cortical networks (e.g. visual or auditory networks), they are systematically outperformed by our multimodal model in high-level associative cortices. Currently applied to perception and comprehension, our approach paves the way towards building an integrative model of representations in the human brain. Our code is available at https://github.com/facebookresearch/algonauts-2025.