A Penny for Your Thoughts: Decoding Speech from Inexpensive Brain Signals
This work addresses brain-to-speech decoding for brain-computer interface applications, but it is incremental as it builds on existing state-of-the-art methods with minor architectural changes.
The paper tackled the problem of decoding speech from EEG brain signals by training a neural network with a contrastive CLIP loss to align EEG embeddings with speech model embeddings, resulting in improvements such as a 1.87% reduction in word error rate (WER) from a dual-path RNN modification.
We explore whether neural networks can decode brain activity into speech by mapping EEG recordings to audio representations. Using EEG data recorded as subjects listened to natural speech, we train a model with a contrastive CLIP loss to align EEG-derived embeddings with embeddings from a pre-trained transformer-based speech model. Building on the state-of-the-art EEG decoder from Meta, we introduce three architectural modifications: (i) subject-specific attention layers (+0.15% WER improvement), (ii) personalized spatial attention (+0.45%), and (iii) a dual-path RNN with attention (-1.87%). Two of the three modifications improved performance, highlighting the promise of personalized architectures for brain-to-speech decoding and applications in brain-computer interfaces.