Towards Dynamic Neural Communication and Speech Neuroprosthesis Based on Viseme Decoding
This work advances speech neuroprosthesis for patients and neural communication tools, though it is incremental as it builds on existing decoding methods with a focus on visemes.
The study tackled the challenge of decoding coherent visual speech intentions from non-invasive brain signals by developing a diffusion model-based framework to reconstruct lip movements from visemes, successfully bridging brain signals and dynamic visual interfaces for neural communication.
Decoding text, speech, or images from human neural signals holds promising potential both as neuroprosthesis for patients and as innovative communication tools for general users. Although neural signals contain various information on speech intentions, movements, and phonetic details, generating informative outputs from them remains challenging, with mostly focusing on decoding short intentions or producing fragmented outputs. In this study, we developed a diffusion model-based framework to decode visual speech intentions from speech-related non-invasive brain signals, to facilitate face-to-face neural communication. We designed an experiment to consolidate various phonemes to train visemes of each phoneme, aiming to learn the representation of corresponding lip formations from neural signals. By decoding visemes from both isolated trials and continuous sentences, we successfully reconstructed coherent lip movements, effectively bridging the gap between brain signals and dynamic visual interfaces. The results highlight the potential of viseme decoding and talking face reconstruction from human neural signals, marking a significant step toward dynamic neural communication systems and speech neuroprosthesis for patients.