Scalable Frameworks for Real-World Audio-Visual Speech Recognition
It addresses the problem of unreliable AVSR in noisy settings for practical deployment, but it is incremental as it builds on existing methods without claiming a paradigm shift.
This dissertation tackles the performance degradation of Audio-Visual Speech Recognition (AVSR) systems in real-world environments with unpredictable noise and interference, proposing a hierarchical approach at representation, architecture, and system levels to build a robust and scalable system.
The practical deployment of Audio-Visual Speech Recognition (AVSR) systems is fundamentally challenged by significant performance degradation in real-world environments, characterized by unpredictable acoustic noise and visual interference. This dissertation posits that a systematic, hierarchical approach is essential to overcome these challenges, achieving the robust scalability at the representation, architecture, and system levels. At the representation level, we investigate methods for building a unified model that learns audio-visual features inherently robust to diverse real-world corruptions, thereby enabling generalization to new environments without specialized modules. To address architectural scalability, we explore how to efficiently expand model capacity while ensuring the adaptive and reliable use of multimodal inputs, developing a framework that intelligently allocates computational resources based on the input characteristics. Finally, at the system level, we present methods to expand the system's functionality through modular integration with large-scale foundation models, leveraging their powerful cognitive and generative capabilities to maximize final recognition accuracy. By systematically providing solutions at each of these three levels, this dissertation aims to build a next-generation, robust, and scalable AVSR system with high reliability in real-world applications.