GLAP: General contrastive audio-text pretraining across domains and languages
This work addresses the problem of multilingual audio-text understanding for applications like retrieval and classification, representing an incremental improvement by extending existing methods to new domains and languages.
The paper tackles the limitation of existing audio-text pretraining methods to English by introducing GLAP, a model that expands to multilingual and multi-domain capabilities, achieving competitive results on standard benchmarks like Clotho and AudioCaps while significantly outperforming previous methods in speech retrieval and classification tasks across 50 languages.
Contrastive Language Audio Pretraining (CLAP) is a widely-used method to bridge the gap between audio and text domains. Current CLAP methods enable sound and music retrieval in English, ignoring multilingual spoken content. To address this, we introduce general language audio pretraining (GLAP), which expands CLAP with multilingual and multi-domain abilities. GLAP demonstrates its versatility by achieving competitive performance on standard audio-text retrieval benchmarks like Clotho and AudioCaps, while significantly surpassing existing methods in speech retrieval and classification tasks. Additionally, GLAP achieves strong results on widely used sound-event zero-shot benchmarks, while simultaneously outperforming previous methods on speech content benchmarks. Further keyword spotting evaluations across 50 languages emphasize GLAP's advanced multilingual capabilities. Finally, multilingual sound and music understanding is evaluated across four languages. Checkpoints and Source: https://github.com/xiaomi-research/dasheng-glap.