Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading
This addresses performance limitations in lip-reading systems, which have applications in assistive technology and human-computer interaction, though it appears to be an incremental improvement over existing methods.
The paper tackles the exposure bias and optimization-evaluation inconsistency problems in sequence-to-sequence lip-reading models by proposing a pseudo-convolutional policy gradient method that incorporates character error rate as a reward and uses contextual information for robust optimization. The results show significant improvements, achieving state-of-the-art or competitive accuracy on word-level and sentence-level benchmarks.
Lip-reading aims to infer the speech content from the lip movement sequence and can be seen as a typical sequence-to-sequence (seq2seq) problem which translates the input image sequence of lip movements to the text sequence of the speech content. However, the traditional learning process of seq2seq models always suffers from two problems: the exposure bias resulted from the strategy of "teacher-forcing", and the inconsistency between the discriminative optimization target (usually the cross-entropy loss) and the final evaluation metric (usually the character/word error rate). In this paper, we propose a novel pseudo-convolutional policy gradient (PCPG) based method to address these two problems. On the one hand, we introduce the evaluation metric (refers to the character error rate in this paper) as a form of reward to optimize the model together with the original discriminative target. On the other hand, inspired by the local perception property of convolutional operation, we perform a pseudo-convolutional operation on the reward and loss dimension, so as to take more context around each time step into account to generate a robust reward and loss for the whole optimization. Finally, we perform a thorough comparison and evaluation on both the word-level and sentence-level benchmarks. The results show a significant improvement over other related methods, and report either a new state-of-the-art performance or a competitive accuracy on all these challenging benchmarks, which clearly proves the advantages of our approach.