BrainVista: Modeling Naturalistic Brain Dynamics as Multimodal Next-Token Prediction
This work addresses the problem of realistic neural simulation for researchers in neuroscience and AI, though it appears incremental as it builds on existing multimodal autoregressive frameworks.
The paper tackled modeling naturalistic brain dynamics as multimodal next-token prediction, introducing BrainVista to address challenges in simulating causal evolution, and achieved state-of-the-art fMRI encoding with improvements of 36.0% and 33.3% in pattern correlation over baselines on key datasets.
Naturalistic fMRI characterizes the brain as a dynamic predictive engine driven by continuous sensory streams. However, modeling the causal forward evolution in realistic neural simulation is impeded by the timescale mismatch between multimodal inputs and the complex topology of cortical networks. To address these challenges, we introduce BrainVista, a multimodal autoregressive framework designed to model the causal evolution of brain states. BrainVista incorporates Network-wise Tokenizers to disentangle system-specific dynamics and a Spatial Mixer Head that captures inter-network information flow without compromising functional boundaries. Furthermore, we propose a novel Stimulus-to-Brain (S2B) masking mechanism to synchronize high-frequency sensory stimuli with hemodynamically filtered signals, enabling strict, history-only causal conditioning. We validate our framework on Algonauts 2025, CineBrain, and HAD, achieving state-of-the-art fMRI encoding performance. In long-horizon rollout settings, our model yields substantial improvements over baselines, increasing pattern correlation by 36.0\% and 33.3\% on relative to the strongest baseline Algonauts 2025 and CineBrain, respectively.