Mapping fMRI Signal and Image Stimuli in an Artificial Neural Network Latent Space: Bringing Artificial and Natural Minds Together
This preliminary work addresses the challenge of decoding stimuli from fMRI data, which could aid in understanding neural representations and improving ANN interpretability, but it is incremental as it only explores feasibility without concrete advancements.
The study investigated whether latent space representations of fMRI data and visual stimuli share common information, but found that the latent spaces of an autoencoder and a vision transformer appeared different, with results being inconclusive and requiring further research.
The goal of this study is to investigate whether latent space representations of visual stimuli and fMRI data share common information. Decoding and reconstructing stimuli from fMRI data remains a challenge in AI and neuroscience, with significant implications for understanding neural representations and improving the interpretability of Artificial Neural Networks (ANNs). In this preliminary study, we investigate the feasibility of such reconstruction by examining the similarity between the latent spaces of one autoencoder (AE) and one vision transformer (ViT) trained on fMRI and image data, respectively. Using representational similarity analysis (RSA), we found that the latent spaces of the two domains appear different. However, these initial findings are inconclusive, and further research is needed to explore this relationship more thoroughly.