SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model
This work addresses the challenge of data scarcity in speech processing for researchers and practitioners by reducing reliance on expensive transcriptions.
The paper tackles the problem of costly transcribed speech data by proposing SpeechCLIP, a framework that bridges speech and text through images to enhance speech models without transcriptions, achieving state-of-the-art performance on image-speech retrieval and enabling zero-shot speech-text retrieval.
Data-driven speech processing models usually perform well with a large amount of text supervision, but collecting transcribed speech data is costly. Therefore, we propose SpeechCLIP, a novel framework bridging speech and text through images to enhance speech models without transcriptions. We leverage state-of-the-art pre-trained HuBERT and CLIP, aligning them via paired images and spoken captions with minimal fine-tuning. SpeechCLIP outperforms prior state-of-the-art on image-speech retrieval and performs zero-shot speech-text retrieval without direct supervision from transcriptions. Moreover, SpeechCLIP can directly retrieve semantically related keywords from speech.