Dynadiff: Single-stage Decoding of Images from Continuously Evolving fMRI
This work addresses the limitation of current brain-to-image decoding methods that collapse temporal dimensions, providing a foundation for time-resolved decoding, which is incremental in simplifying training and improving performance.
The paper tackles the problem of reconstructing images from continuously evolving fMRI recordings by introducing Dynadiff, a single-stage diffusion model that outperforms state-of-the-art models on time-resolved fMRI signals, particularly in high-level semantic image reconstruction metrics.
Brain-to-image decoding has been recently propelled by the progress in generative AI models and the availability of large ultra-high field functional Magnetic Resonance Imaging (fMRI). However, current approaches depend on complicated multi-stage pipelines and preprocessing steps that typically collapse the temporal dimension of brain recordings, thereby limiting time-resolved brain decoders. Here, we introduce Dynadiff (Dynamic Neural Activity Diffusion for Image Reconstruction), a new single-stage diffusion model designed for reconstructing images from dynamically evolving fMRI recordings. Our approach offers three main contributions. First, Dynadiff simplifies training as compared to existing approaches. Second, our model outperforms state-of-the-art models on time-resolved fMRI signals, especially on high-level semantic image reconstruction metrics, while remaining competitive on preprocessed fMRI data that collapse time. Third, this approach allows a precise characterization of the evolution of image representations in brain activity. Overall, this work lays the foundation for time-resolved brain-to-image decoding.