Audio-visual fine-tuning of audio-only ASR models
This work addresses the challenge of data efficiency and computational cost in audio-visual speech recognition for researchers and practitioners, offering a simpler alternative to complex existing methods.
The paper tackled the problem of reducing the need for large amounts of transcribed audio-visual data in audio-visual automatic speech recognition by proposing a simpler and faster method using audio-only self-supervised learning followed by audio-visual fine-tuning. The result was competitive performance with state-of-the-art methods on the LRS3-TED benchmark, achieving within 0.5% absolute word error rate, while being 12-30x faster to pre-train.
Audio-visual automatic speech recognition (AV-ASR) models are very effective at reducing word error rates on noisy speech, but require large amounts of transcribed AV training data. Recently, audio-visual self-supervised learning (SSL) approaches have been developed to reduce this dependence on transcribed AV data, but these methods are quite complex and computationally expensive. In this work, we propose replacing these expensive AV-SSL methods with a simple and fast \textit{audio-only} SSL method, and then performing AV supervised fine-tuning. We show that this approach is competitive with state-of-the-art (SOTA) AV-SSL methods on the LRS3-TED benchmark task (within 0.5% absolute WER), while being dramatically simpler and more efficient (12-30x faster to pre-train). Furthermore, we show we can extend this approach to convert a SOTA audio-only ASR model into an AV model. By doing so, we match SOTA AV-SSL results, even though no AV data was used during pre-training.