AVATAR submission to the Ego4D AV Transcription Challenge
This work addresses audio-visual speech recognition for egocentric video data, but it is incremental as it applies an existing state-of-the-art method to a new challenge.
The authors tackled the problem of audio-visual speech transcription in the Ego4D challenge by using the AVATAR model, achieving a word error rate of 68.40 and outperforming the baseline by 43.7% to win the challenge.
In this report, we describe our submission to the Ego4D AudioVisual (AV) Speech Transcription Challenge 2022. Our pipeline is based on AVATAR, a state of the art encoder-decoder model for AV-ASR that performs early fusion of spectrograms and RGB images. We describe the datasets, experimental settings and ablations. Our final method achieves a WER of 68.40 on the challenge test set, outperforming the baseline by 43.7%, and winning the challenge.