Deep Multimodal Learning for Audio-Visual Speech Recognition
This work improves speech recognition accuracy by integrating visual cues, which is incremental as it builds on existing multimodal fusion methods.
The paper tackles audio-visual speech recognition by fusing speech and visual modalities, achieving a phone error rate (PER) of 34.03% through a combination of fusion models and a bilinear network, compared to 41% with audio alone.
In this paper, we present methods in deep multimodal learning for fusing speech and visual modalities for Audio-Visual Automatic Speech Recognition (AV-ASR). First, we study an approach where uni-modal deep networks are trained separately and their final hidden layers fused to obtain a joint feature space in which another deep network is built. While the audio network alone achieves a phone error rate (PER) of $41\%$ under clean condition on the IBM large vocabulary audio-visual studio dataset, this fusion model achieves a PER of $35.83\%$ demonstrating the tremendous value of the visual channel in phone classification even in audio with high signal to noise ratio. Second, we present a new deep network architecture that uses a bilinear softmax layer to account for class specific correlations between modalities. We show that combining the posteriors from the bilinear networks with those from the fused model mentioned above results in a further significant phone error rate reduction, yielding a final PER of $34.03\%$.