Scaling Vision Transformers for Functional MRI with Flat Maps
This work addresses the problem of representing fMRI data for deep learning in neuroscience, offering a novel approach that could support foundation models for brain analysis, though it appears incremental in adapting existing methods to a new modality.
The researchers tackled the challenge of adapting Vision Transformers to functional MRI (fMRI) data by transforming 4D volumetric fMRI into 2D flat map videos, training on 2.3K hours of data, and observed that masked modeling performance improved with dataset size following a power scaling law, enabling fine-grained state and trait decoding.
A key question for adapting modern deep learning architectures to functional MRI (fMRI) is how to represent the data for model input. To bridge the modality gap between fMRI and natural images, we transform the 4D volumetric fMRI data into videos of 2D fMRI activity flat maps. We train Vision Transformers on 2.3K hours of fMRI flat map videos from the Human Connectome Project using the spatiotemporal masked autoencoder (MAE) framework. We observe that masked fMRI modeling performance improves with dataset size according to a strict power scaling law. Downstream classification benchmarks show that our model learns rich representations supporting both fine-grained state decoding across subjects, as well as subject-specific trait decoding across changes in brain state. This work is part of an ongoing open science project to build foundation models for fMRI data. Our code and datasets are available at https://github.com/MedARC-AI/fmri-fm.