ASR is all you need: cross-modal distillation for lip reading
This addresses the problem of data scarcity for lip reading systems by enabling use of unlabeled video, but it is incremental as it builds on existing distillation and ASR methods.
The paper tackles training visual speech recognition models without human-annotated ground truth by distilling from an ASR model trained on large-scale audio data, achieving state-of-the-art results on LRS2 and LRS3 datasets using only publicly available data.
The goal of this work is to train strong models for visual speech recognition without requiring human annotated ground truth data. We achieve this by distilling from an Automatic Speech Recognition (ASR) model that has been trained on a large-scale audio-only corpus. We use a cross-modal distillation method that combines Connectionist Temporal Classification (CTC) with a frame-wise cross-entropy loss. Our contributions are fourfold: (i) we show that ground truth transcriptions are not necessary to train a lip reading system; (ii) we show how arbitrary amounts of unlabelled video data can be leveraged to improve performance; (iii) we demonstrate that distillation significantly speeds up training; and, (iv) we obtain state-of-the-art results on the challenging LRS2 and LRS3 datasets for training only on publicly available data.