VIBE: Video-Input Brain Encoder for fMRI Response Modeling
This work addresses brain activity modeling for neuroscience applications, representing an incremental improvement with a specific architecture.
The paper tackles the problem of predicting fMRI activity from multi-modal video inputs, achieving mean parcel-wise Pearson correlations of 0.3225 on in-distribution data and 0.2125 on out-of-distribution films.
We present VIBE, a two-stage Transformer that fuses multi-modal video, audio, and text features to predict fMRI activity. Representations from open-source models (Qwen2.5, BEATs, Whisper, SlowFast, V-JEPA) are merged by a modality-fusion transformer and temporally decoded by a prediction transformer with rotary embeddings. Trained on 65 hours of movie data from the CNeuroMod dataset and ensembled across 20 seeds, VIBE attains mean parcel-wise Pearson correlations of 0.3225 on in-distribution Friends S07 and 0.2125 on six out-of-distribution films. An earlier iteration of the same architecture obtained 0.3198 and 0.2096, respectively, winning Phase-1 and placing second overall in the Algonauts 2025 Challenge.