WaveMind: Towards a Conversational EEG Foundation Model Aligned to Textual and Visual Modalities
This work addresses the problem of EEG interpretation for neuroscience research and general-purpose EEG model development, representing an incremental advance by building on existing multimodal approaches.
The paper tackles the challenge of interpreting EEG signals with multimodal large language models by addressing the mismatch between EEG data and other modalities, proposing a method to map them into a unified semantic space. The result is a model that achieves robust classification accuracy and supports open-ended conversations across four downstream tasks, using a new dataset of 338k instructions.
Electroencephalography (EEG) interpretation using multimodal large language models (MLLMs) offers a novel approach for analyzing brain signals. However, the complex nature of brain activity introduces critical challenges: EEG signals simultaneously encode both cognitive processes and intrinsic neural states, creating a mismatch in EEG paired-data modality that hinders effective cross-modal representation learning. Through a pivot investigation, we uncover complementary relationships between these modalities. Leveraging this insight, we propose mapping EEG signals and their corresponding modalities into a unified semantic space to achieve generalized interpretation. To fully enable conversational capabilities, we further introduce WaveMind-Instruct-338k, the first cross-task EEG dataset for instruction tuning. The resulting model demonstrates robust classification accuracy while supporting flexible, open-ended conversations across four downstream tasks, thereby offering valuable insights for both neuroscience research and the development of general-purpose EEG models.