CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention
This work addresses limitations in EEG analysis for researchers and clinicians by providing a more efficient and reproducible foundation model, though it is incremental in improving existing self-supervised methods.
The authors tackled the problem of inefficient and suboptimal EEG signal modeling in self-supervised learning by introducing CEReBrO, a compact encoder that uses alternating attention, achieving 2x speed improvement and 6x less memory usage while setting new benchmarks in emotion and seizure detection tasks.
Electroencephalograph (EEG) is a crucial tool for studying brain activity. Recently, self-supervised learning methods leveraging large unlabeled datasets have emerged as a potential solution to the scarcity of widely available annotated EEG data. However, current methods suffer from at least one of the following limitations: i) sub-optimal EEG signal modeling, ii) model sizes in the hundreds of millions of trainable parameters, and iii) reliance on private datasets and/or inconsistent public benchmarks, hindering reproducibility. To address these challenges, we introduce a Compact Encoder for Representations of Brain Oscillations using alternating attention (CEReBrO), a new small EEG foundation model. Our tokenization scheme represents EEG signals at a per-channel patch granularity. We propose an alternating attention mechanism that jointly models intra-channel temporal dynamics and inter-channel spatial correlations, achieving 2x speed improvement with 6x less memory required compared to standard self-attention. We present several model sizes ranging from 3.6 million to 85 million parameters. Pre-trained on over 20,000 hours of publicly available scalp EEG recordings with diverse channel configurations, our models set new benchmarks in emotion detection and seizure detection tasks, with competitive performance in anomaly classification and gait prediction. This validates our models' effectiveness and efficiency.