EEG-CLIP : Learning EEG representations from natural language descriptions
This work addresses the need for task-agnostic EEG analysis, potentially easing diverse decoding tasks through zero-shot capabilities, though it is incremental as it adapts existing vision-language methods to EEG.
The authors tackled the problem of learning general EEG representations by developing EEG-CLIP, a contrastive learning framework that aligns EEG time series with clinical text descriptions, enabling versatile decoding in few-shot and zero-shot settings with non-trivial alignment results.
Deep networks for electroencephalogram (EEG) decoding are often only trained to solve one specific task, such as pathology or age decoding. A more general task-agnostic approach is to train deep networks to match a (clinical) EEG recording to its corresponding textual medical report and vice versa. This approach was pioneered in the computer vision domain matching images and their text captions and subsequently allowed to do successful zero-shot decoding using textual class prompts. In this work, we follow this approach and develop a contrastive learning framework, EEG-CLIP, that aligns the EEG time series and the descriptions of the corresponding clinical text in a shared embedding space. We investigated its potential for versatile EEG decoding, evaluating performance in a range of few-shot and zero-shot settings. Overall, we show that EEG-CLIP manages to non-trivially align text and EEG representations. Our work presents a promising approach to learn general EEG representations, which could enable easier analyses of diverse decoding questions through zero-shot decoding or training task-specific models from fewer training examples. The code for reproducing our results is available at https://github.com/tidiane-camaret/EEGClip