Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading
This work addresses the problem of making automatic speech recognition more robust and accurate for applications in noisy environments, though it appears incremental as it builds on prior DNN approaches.
The paper tackles the problem of audio-visual speech recognition (AVSR) by proposing an Auxiliary Multimodal LSTM (am-LSTM) model to address shortcomings in existing DNN methods, such as imbalanced modal and temporal fusion and non-end-to-end architectures, resulting in improved performance over traditional and other DNN models on three datasets.
The Aduio-visual Speech Recognition (AVSR) which employs both the video and audio information to do Automatic Speech Recognition (ASR) is one of the application of multimodal leaning making ASR system more robust and accuracy. The traditional models usually treated AVSR as inference or projection but strict prior limits its ability. As the revival of deep learning, Deep Neural Networks (DNN) becomes an important toolkit in many traditional classification tasks including ASR, image classification, natural language processing. Some DNN models were used in AVSR like Multimodal Deep Autoencoders (MDAEs), Multimodal Deep Belief Network (MDBN) and Multimodal Deep Boltzmann Machine (MDBM) that actually work better than traditional methods. However, such DNN models have several shortcomings: (1) They don't balance the modal fusion and temporal fusion, or even haven't temporal fusion; (2)The architecture of these models isn't end-to-end, the training and testing getting cumbersome. We propose a DNN model, Auxiliary Multimodal LSTM (am-LSTM), to overcome such weakness. The am-LSTM could be trained and tested once, moreover easy to train and preventing overfitting automatically. The extensibility and flexibility are also take into consideration. The experiments show that am-LSTM is much better than traditional methods and other DNN models in three datasets.