End-to-End Visual Speech Recognition for Small-Scale Datasets
This addresses the challenge of limited data availability in visual speech recognition, though it is incremental as it builds on existing deep learning approaches.
The paper tackles the problem of visual speech recognition on small-scale datasets by proposing an end-to-end system using fully-connected layers and LSTM networks, achieving absolute improvements of 0.6% to 11.4% over state-of-the-art on four databases.
Visual speech recognition models traditionally consist of two stages, feature extraction and classification. Several deep learning approaches have been recently presented aiming to replace the feature extraction stage by automatically extracting features from mouth images. However, research on joint learning of features and classification remains limited. In addition, most of the existing methods require large amounts of data in order to achieve state-of-the-art performance, otherwise they under-perform. In this work, we present an end-to-end visual speech recognition system based on fully-connected layers and Long-Short Memory (LSTM) networks which is suitable for small-scale datasets. The model consists of two streams which extract features directly from the mouth and difference images, respectively. The temporal dynamics in each stream are modelled by a Bidirectional LSTM (BLSTM) and the fusion of the two streams takes place via another BLSTM. An absolute improvement of 0.6%, 3.4%, 3.9%, 11.4% over the state-of-the-art is reported on the OuluVS2, CUAVE, AVLetters and AVLetters2 databases, respectively.