Leveraging Audio-Visual Data to Reduce the Multilingual Gap in Self-Supervised Speech Models
This addresses the problem of underperformance in multilingual speech models for applications like speech recognition, though it is incremental as it builds on existing SSL methods.
The paper tackled the performance gap between monolingual and multilingual self-supervised speech models by introducing visual grounding, reducing the gap in zero-shot phonetic discrimination from 31.5% to 8.04%.
Self-supervised learning (SSL) has made significant advances in speech representation learning. Models like wav2vec 2.0 and HuBERT have achieved state-of-the-art results in tasks such as speech recognition, particularly in monolingual settings. However, multilingual SSL models tend to underperform their monolingual counterparts on each individual language, especially in multilingual scenarios with few languages such as the bilingual setting. In this work, we investigate a novel approach to reduce this performance gap by introducing limited visual grounding into bilingual speech SSL models. Our results show that visual grounding benefits both monolingual and bilingual models, with especially pronounced gains for the latter, reducing the multilingual performance gap on zero-shot phonetic discrimination from 31.5% for audio-only models to 8.04% with grounding.