Visual Speech Language Models
This work addresses the challenge of improving lipreading systems for applications like assistive technology, but it appears incremental as it focuses on comparing existing units rather than introducing a new method.
The paper tackled the problem of visual speech language models underperforming due to visual co-articulation effects, and found that comparing visemes, phonemes, and words helps optimize lipreading models to observe their limitations.
Language models (LM) are very powerful in lipreading systems. Language models built upon the ground truth utterances of datasets learn grammar and structure rules of words and sentences (the latter in the case of continuous speech). However, visual co-articulation effects in visual speech signals damage the performance of visual speech LM's as visually, people do not utter what the language model expects. These models are commonplace but while higher-order N-gram LM's may improve classification rates, the cost of this model is disproportionate to the common goal of developing more accurate classifiers. So we compare which unit would best optimize a lipreading (visual speech) LM to observe their limitations. We compare three units; visemes (visual speech units) \cite{lan2010improving}, phonemes (audible speech units), and words.