LLM4Brain: Training a Large Language Model for Brain Video Understanding
This addresses the challenge of brain video understanding for neuroscience and AI applications, but appears incremental as it builds on existing LLM and adaptation techniques.
The study tackled the problem of decoding visual-semantic information from fMRI brain signals across subjects by introducing an LLM-based approach with fine-tuning and domain adaptation, achieving good results on semantic metrics and similarity to ground-truth.
Decoding visual-semantic information from brain signals, such as functional MRI (fMRI), across different subjects poses significant challenges, including low signal-to-noise ratio, limited data availability, and cross-subject variability. Recent advancements in large language models (LLMs) show remarkable effectiveness in processing multimodal information. In this study, we introduce an LLM-based approach for reconstructing visual-semantic information from fMRI signals elicited by video stimuli. Specifically, we employ fine-tuning techniques on an fMRI encoder equipped with adaptors to transform brain responses into latent representations aligned with the video stimuli. Subsequently, these representations are mapped to textual modality by LLM. In particular, we integrate self-supervised domain adaptation methods to enhance the alignment between visual-semantic information and brain responses. Our proposed method achieves good results using various quantitative semantic metrics, while yielding similarity with ground-truth information.