Dynamic Neural Communication: Convergence of Computer Vision and Brain-Computer Interface
This work addresses the problem of enabling more informative and dynamic communication for individuals with speech impairments by integrating computer vision and brain-computer interface technologies, though it appears incremental as it builds on existing methods.
The study tackled the challenge of decoding dynamic speech intentions from neural signals by introducing a method that captures user intentions and decodes visemes in short time steps to produce dynamic visual outputs, resulting in the potential to rapidly reconstruct lip movements during natural speech attempts.
Interpreting human neural signals to decode static speech intentions such as text or images and dynamic speech intentions such as audio or video is showing great potential as an innovative communication tool. Human communication accompanies various features, such as articulatory movements, facial expressions, and internal speech, all of which are reflected in neural signals. However, most studies only generate short or fragmented outputs, while providing informative communication by leveraging various features from neural signals remains challenging. In this study, we introduce a dynamic neural communication method that leverages current computer vision and brain-computer interface technologies. Our approach captures the user's intentions from neural signals and decodes visemes in short time steps to produce dynamic visual outputs. The results demonstrate the potential to rapidly capture and reconstruct lip movements during natural speech attempts from human neural signals, enabling dynamic neural communication through the convergence of computer vision and brain--computer interface.