MindCine: Multimodal EEG-to-Video Reconstruction with Large-Scale Pretrained Models
This work addresses EEG-to-video reconstruction for neuroscience and brain-computer interfaces, but it is incremental as it builds on existing methods by incorporating multimodal learning and pretrained models.
The authors tackled the problem of reconstructing videos from EEG signals by addressing single-modality limitations and data scarcity, achieving state-of-the-art performance with high-fidelity reconstructions as demonstrated in experiments.
Reconstructing human dynamic visual perception from electroencephalography (EEG) signals is of great research significance since EEG's non-invasiveness and high temporal resolution. However, EEG-to-video reconstruction remains challenging due to: 1) Single Modality: existing studies solely align EEG signals with the text modality, which ignores other modalities and are prone to suffer from overfitting problems; 2) Data Scarcity: current methods often have difficulty training to converge with limited EEG-video data. To solve the above problems, we propose a novel framework MindCine to achieve high-fidelity video reconstructions on limited data. We employ a multimodal joint learning strategy to incorporate beyond-text modalities in the training stage and leverage a pre-trained large EEG model to relieve the data scarcity issue for decoding semantic information, while a Seq2Seq model with causal attention is specifically designed for decoding perceptual information. Extensive experiments demonstrate that our model outperforms state-of-the-art methods both qualitatively and quantitatively. Additionally, the results underscore the effectiveness of the complementary strengths of different modalities and demonstrate that leveraging a large-scale EEG model can further enhance reconstruction performance by alleviating the challenges associated with limited data.