Brain-to-Text Decoding: A Non-invasive Approach via Typing
This work addresses the need for safe brain-computer interfaces for non-communicating patients by narrowing the gap with invasive methods, though it is incremental as it builds on existing neuroprosthesis concepts.
The researchers tackled the problem of decoding sentences from brain activity non-invasively, introducing Brain2Qwerty, which achieved an average character-error-rate of 32% with MEG and 19% for the best participants, outperforming EEG and enabling perfect decoding of some new sentences.
Modern neuroprostheses can now restore communication in patients who have lost the ability to speak or move. However, these invasive devices entail risks inherent to neurosurgery. Here, we introduce a non-invasive method to decode the production of sentences from brain activity and demonstrate its efficacy in a cohort of 35 healthy volunteers. For this, we present Brain2Qwerty, a new deep learning architecture trained to decode sentences from either electro- (EEG) or magneto-encephalography (MEG), while participants typed briefly memorized sentences on a QWERTY keyboard. With MEG, Brain2Qwerty reaches, on average, a character-error-rate (CER) of 32% and substantially outperforms EEG (CER: 67%). For the best participants, the model achieves a CER of 19%, and can perfectly decode a variety of sentences outside of the training set. While error analyses suggest that decoding depends on motor processes, the analysis of typographical errors suggests that it also involves higher-level cognitive factors. Overall, these results narrow the gap between invasive and non-invasive methods and thus open the path for developing safe brain-computer interfaces for non-communicating patients.