Neural Decoding of Overt Speech from ECoG Using Vision Transformers and Contrastive Representation Learning
This work addresses speech reconstruction for people with severe paralysis using BCIs, presenting a novel approach with a fully implantable system, though it is incremental in optimizing neural decoders for surface ECoG.
The paper tackles the challenge of reconstructing speech from surface electrocorticographic (ECoG) recordings in a streaming mode by directly regressing cortical signals into acoustic speech, achieving results with a fully implantable wireless epidural system for potential long-term use.
Speech Brain Computer Interfaces (BCIs) offer promising solutions to people with severe paralysis unable to communicate. A number of recent studies have demonstrated convincing reconstruction of intelligible speech from surface electrocorticographic (ECoG) or intracortical recordings by predicting a series of phonemes or words and using downstream language models to obtain meaningful sentences. A current challenge is to reconstruct speech in a streaming mode by directly regressing cortical signals into acoustic speech. While this has been achieved recently using intracortical data, further work is needed to obtain comparable results with surface ECoG recordings. In particular, optimizing neural decoders becomes critical in this case. Here we present an offline speech decoding pipeline based on an encoder-decoder deep neural architecture, integrating Vision Transformers and contrastive learning to enhance the direct regression of speech from ECoG signals. The approach is evaluated on two datasets, one obtained with clinical subdural electrodes in an epileptic patient, and another obtained with the fully implantable WIMAGINE epidural system in a participant of a motor BCI trial. To our knowledge this presents a first attempt to decode speech from a fully implantable and wireless epidural recording system offering perspectives for long-term use.