Lip reading using external viseme decoding
This work addresses the challenge of accurate speech recognition from lip movements for applications like assistive technology, but it is incremental as it builds on existing two-stage approaches.
The paper tackles lip-reading by splitting video-to-character conversion into two stages: video to viseme and viseme to character, using external text data for mapping. This method reduces word error rate by 4% on the LRS2 dataset compared to a standard sequence-to-sequence model.
Lip-reading is the operation of recognizing speech from lip movements. This is a difficult task because the movements of the lips when pronouncing the words are similar for some of them. Viseme is used to describe lip movements during a conversation. This paper aims to show how to use external text data (for viseme-to-character mapping) by dividing video-to-character into two stages, namely converting video to viseme, and then converting viseme to character by using separate models. Our proposed method improves word error rate by 4\% compared to the normal sequence to sequence lip-reading model on the BBC-Oxford Lip Reading Sentences 2 (LRS2) dataset.