Vid2speech: Speech Reconstruction from Silent Video
This addresses the difficult task of speechreading for applications like assistive technology or surveillance, but it is incremental as it builds on existing CNN methods.
The paper tackles the problem of reconstructing intelligible speech from silent video of a speaking person, achieving state-of-the-art word intelligibility on the GRID dataset and showing promising results for out-of-vocabulary words.
Speechreading is a notoriously difficult task for humans to perform. In this paper we present an end-to-end model based on a convolutional neural network (CNN) for generating an intelligible acoustic speech signal from silent video frames of a speaking person. The proposed CNN generates sound features for each frame based on its neighboring frames. Waveforms are then synthesized from the learned speech features to produce intelligible speech. We show that by leveraging the automatic feature learning capabilities of a CNN, we can obtain state-of-the-art word intelligibility on the GRID dataset, and show promising results for learning out-of-vocabulary (OOV) words.