AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR
This addresses the need for robust speech recognition in noisy environments by enabling zero-shot AV-ASR with minimal data and training, though it is incremental as it builds on existing frozen speech models.
The paper tackles the problem of training multimodal models for audiovisual automatic speech recognition (AV-ASR) without large labeled datasets by introducing AVFormer, a method that injects visual embeddings into frozen speech models using lightweight adaptors, achieving state-of-the-art zero-shot results on three benchmarks (How2, VisSpeech, and Ego4D) while maintaining performance on audio-only tasks like LibriSpeech.
Audiovisual automatic speech recognition (AV-ASR) aims to improve the robustness of a speech recognition system by incorporating visual information. Training fully supervised multimodal models for this task from scratch, however is limited by the need for large labelled audiovisual datasets (in each downstream domain of interest). We present AVFormer, a simple method for augmenting audio-only models with visual information, at the same time performing lightweight domain adaptation. We do this by (i) injecting visual embeddings into a frozen ASR model using lightweight trainable adaptors. We show that these can be trained on a small amount of weakly labelled video data with minimum additional training time and parameters. (ii) We also introduce a simple curriculum scheme during training which we show is crucial to enable the model to jointly process audio and visual information effectively; and finally (iii) we show that our model achieves state of the art zero-shot results on three different AV-ASR benchmarks (How2, VisSpeech and Ego4D), while also crucially preserving decent performance on traditional audio-only speech recognition benchmarks (LibriSpeech). Qualitative results show that our model effectively leverages visual information for robust speech recognition.