InfoSyncNet: Information Synchronization Temporal Convolutional Network for Visual Speech Recognition
This addresses visual speech recognition for assistive and augmented reality applications, representing an incremental improvement with specific gains.
The paper tackles the problem of accurately mapping lip movements in silent videos to words for applications like Assistive Technology and Augmented Reality, achieving state-of-the-art accuracies of 92.0% on LRW and 60.7% on LRW1000 datasets.
Estimating spoken content from silent videos is crucial for applications in Assistive Technology (AT) and Augmented Reality (AR). However, accurately mapping lip movement sequences in videos to words poses significant challenges due to variability across sequences and the uneven distribution of information within each sequence. To tackle this, we introduce InfoSyncNet, a non-uniform sequence modeling network enhanced by tailored data augmentation techniques. Central to InfoSyncNet is a non-uniform quantization module positioned between the encoder and decoder, enabling dynamic adjustment to the network's focus and effectively handling the natural inconsistencies in visual speech data. Additionally, multiple training strategies are incorporated to enhance the model's capability to handle variations in lighting and the speaker's orientation. Comprehensive experiments on the LRW and LRW1000 datasets confirm the superiority of InfoSyncNet, achieving new state-of-the-art accuracies of 92.0% and 60.7% Top-1 ACC. The code is available for download (see comments).