DreamCatcher: Revealing the Language of the Brain with fMRI using GPT Embedding
This work addresses the challenge of interpreting brain activity for visual processing, with potential applications in cognitive neuroscience and human-computer interaction, but it appears incremental as it builds on existing fMRI-to-image reconstruction methods.
The authors tackled the problem of understanding visual perception by proposing fMRI captioning, where captions are generated from fMRI data, and introduced DreamCatcher, a framework that demonstrated strong performance in this task.
The human brain possesses remarkable abilities in visual processing, including image recognition and scene summarization. Efforts have been made to understand the cognitive capacities of the visual brain, but a comprehensive understanding of the underlying mechanisms still needs to be discovered. Advancements in brain decoding techniques have led to sophisticated approaches like fMRI-to-Image reconstruction, which has implications for cognitive neuroscience and medical imaging. However, challenges persist in fMRI-to-image reconstruction, such as incorporating global context and contextual information. In this article, we propose fMRI captioning, where captions are generated based on fMRI data to gain insight into the neural correlates of visual perception. This research presents DreamCatcher, a novel framework for fMRI captioning. DreamCatcher consists of the Representation Space Encoder (RSE) and the RevEmbedding Decoder, which transform fMRI vectors into a latent space and generate captions, respectively. We evaluated the framework through visualization, dataset training, and testing on subjects, demonstrating strong performance. fMRI-based captioning has diverse applications, including understanding neural mechanisms, Human-Computer Interaction, and enhancing learning and training processes.