SimulLR: Simultaneous Lip Reading Transducer with Attention-Guided Adaptive Memory
This work addresses the need for real-time lip reading in applications like assistive technologies or surveillance, offering a novel approach to overcome the constraint of full-video access, though it builds incrementally on existing transducer and attention methods.
The paper tackles the problem of simultaneous lip reading, where predictions are generated in real-time without requiring the full video, by proposing SimulLR, a transducer-based model with attention-guided adaptive memory, achieving a translation speedup of 9.10× compared to state-of-the-art non-simultaneous methods while maintaining competitive accuracy.
Lip reading, aiming to recognize spoken sentences according to the given video of lip movements without relying on the audio stream, has attracted great interest due to its application in many scenarios. Although prior works that explore lip reading have obtained salient achievements, they are all trained in a non-simultaneous manner where the predictions are generated requiring access to the full video. To breakthrough this constraint, we study the task of simultaneous lip reading and devise SimulLR, a simultaneous lip Reading transducer with attention-guided adaptive memory from three aspects: (1) To address the challenge of monotonic alignments while considering the syntactic structure of the generated sentences under simultaneous setting, we build a transducer-based model and design several effective training strategies including CTC pre-training, model warm-up and curriculum learning to promote the training of the lip reading transducer. (2) To learn better spatio-temporal representations for simultaneous encoder, we construct a truncated 3D convolution and time-restricted self-attention layer to perform the frame-to-frame interaction within a video segment containing fixed number of frames. (3) The history information is always limited due to the storage in real-time scenarios, especially for massive video data. Therefore, we devise a novel attention-guided adaptive memory to organize semantic information of history segments and enhance the visual representations with acceptable computation-aware latency. The experiments show that the SimulLR achieves the translation speedup 9.10$\times$ compared with the state-of-the-art non-simultaneous methods, and also obtains competitive results, which indicates the effectiveness of our proposed methods.