Seeing the imagined: a latent functional alignment in visual imagery decoding from fMRI data
This work addresses the challenge of decoding mental imagery from fMRI data, which is more difficult than perception decoding, by leveraging semantic structure learned from perception data.
The authors adapted a state-of-the-art perception decoder (DynaDiff) for visual imagery decoding from fMRI by proposing a latent functional alignment method that maps imagery-evoked activity into the pretrained model's conditioning space, combined with a retrieval-based augmentation strategy. This approach consistently improved high-level semantic reconstruction metrics across four subjects, enabling above-chance decoding from multiple cortical regions.
Recent progress in visual brain decoding from fMRI has been enabled by large-scale datasets such as the Natural Scenes Dataset (NSD) and powerful diffusion-based generative models. While current pipelines are primarily optimized for perception, their performance under mental-imagery remains less well understood. In this work, we study how a state-of-the-art (SOTA) perception decoder (DynaDiff) can be adapted to reconstruct imagined content from the Imagery-NSD benchmark. We propose a latent functional alignment approach that maps imagery-evoked activity into the pretrained model's conditioning space, while keeping the remaining components frozen. To mitigate the limited amount of matched imagery-perception supervision, we further introduce a retrieval-based augmentation strategy that selects semantically related NSD perception trials. Across four subjects, latent functional alignment consistently improves high-level semantic reconstruction metrics relative to the frozen pretrained baseline and a voxel-space ridge alignment baseline, and enables above-chance decoding from multiple cortical regions. These results suggest that semantic structure learned from perception can be leveraged to stabilize and improve visual imagery decoding under out-of-distribution conditions.