Joint fMRI Decoding and Encoding with Latent Embedding Alignment
This work addresses the problem of modeling brain-visual relationships for neuroscience and AI researchers, offering a comprehensive but incremental solution by integrating decoding and encoding tasks.
The paper tackles the challenge of bidirectional mapping between brain activity (fMRI signals) and visual stimuli by introducing a unified framework for both decoding (recovering images from fMRI) and encoding (predicting fMRI from images), achieving performance that surpasses existing methods on multiple benchmarks.
The connection between brain activity and corresponding visual stimuli is crucial in comprehending the human brain. While deep generative models have exhibited advancement in recovering brain recordings by generating images conditioned on fMRI signals, accomplishing high-quality generation with consistent semantics continues to pose challenges. Moreover, the prediction of brain activity from visual stimuli remains a formidable undertaking. In this paper, we introduce a unified framework that addresses both fMRI decoding and encoding. Commencing with the establishment of two latent spaces capable of representing and reconstructing fMRI signals and visual images, respectively, we proceed to align the fMRI signals and visual images within the latent space, thereby enabling a bidirectional transformation between the two domains. Our Latent Embedding Alignment (LEA) model concurrently recovers visual stimuli from fMRI signals and predicts brain activity from images within a unified framework. The performance of LEA surpasses that of existing methods on multiple benchmark fMRI decoding and encoding datasets. By integrating fMRI decoding and encoding, LEA offers a comprehensive solution for modeling the intricate relationship between brain activity and visual stimuli.