Cocktail-Party Audio-Visual Speech Recognition
This addresses the challenge of robust speech recognition in real-world noisy environments for applications like communication systems, though it is incremental as it builds on existing AVSR methods.
The study tackled the problem of audio-visual speech recognition in noisy cocktail-party scenarios by introducing a novel dataset and approach, resulting in a 67% relative reduction in word error rate from 119% to 39.2% in extreme noise.
Audio-Visual Speech Recognition (AVSR) offers a robust solution for speech recognition in challenging environments, such as cocktail-party scenarios, where relying solely on audio proves insufficient. However, current AVSR models are often optimized for idealized scenarios with consistently active speakers, overlooking the complexities of real-world settings that include both speaking and silent facial segments. This study addresses this gap by introducing a novel audio-visual cocktail-party dataset designed to benchmark current AVSR systems and highlight the limitations of prior approaches in realistic noisy conditions. Additionally, we contribute a 1526-hour AVSR dataset comprising both talking-face and silent-face segments, enabling significant performance gains in cocktail-party environments. Our approach reduces WER by 67% relative to the state-of-the-art, reducing WER from 119% to 39.2% in extreme noise, without relying on explicit segmentation cues.