QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge
This work addresses the problem of audio-visual fusion for ego-centric video analysis, specifically in the Talking to Me task, with an incremental improvement over baseline methods.
The authors tackled the Ego4D Talking to Me Challenge by proposing a quality-aware audio-visual fusion method that uses separate models for video and audio processing, leveraging face quality scores to filter noise and integrate results, achieving 67.4% mAP on the test set and ranking first on the leaderboard.
This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D Talking to Me (TTM) Challenge 2023. Based on the observation from the TTM task and the provided dataset, we propose to use two separate models to process the input videos and audio. By doing so, we can utilize all the labeled training data, including those without bounding box labels. Furthermore, we leverage the face quality score from a facial landmark prediction model for filtering noisy face input data. The face quality score is also employed in our proposed quality-aware fusion for integrating the results from two branches. With the simple architecture design, our model achieves 67.4% mean average precision (mAP) on the test set, which ranks first on the leaderboard and outperforms the baseline method by a large margin. Code is available at: https://github.com/hsi-che-lin/Ego4D-QuAVF-TTM-CVPR23