Towards a practical lip-to-speech conversion system using deep neural networks and mobile application frontend
This work addresses communication challenges for speaking-impaired individuals by developing a practical mobile-based lip-to-speech system, though it appears incremental as it builds on existing offline solutions.
The paper tackles the problem of lip-to-speech conversion by proposing a system with a deep neural network backend and a mobile application frontend, achieving a top-5 classification accuracy of 74% in initial evaluations to aid speaking-impaired communication.
Articulatory-to-acoustic (forward) mapping is a technique to predict speech using various articulatory acquisition techniques as input (e.g. ultrasound tongue imaging, MRI, lip video). The advantage of lip video is that it is easily available and affordable: most modern smartphones have a front camera. There are already a few solutions for lip-to-speech synthesis, but they mostly concentrate on offline training and inference. In this paper, we propose a system built from a backend for deep neural network training and inference and a fronted as a form of a mobile application. Our initial evaluation shows that the scenario is feasible: a top-5 classification accuracy of 74% is combined with feedback from the mobile application user, making sure that the speaking impaired might be able to communicate with this solution.