BrewCLIP: A Bifurcated Representation Learning Framework for Audio-Visual Retrieval
This work addresses audio-image matching for applications like multimedia retrieval, but it appears incremental as it builds on existing models with a bifurcated design.
The paper tackles the problem of audio-visual retrieval by investigating whether non-textual information in speech, such as accent and mood, can improve matching performance, and it achieves a substantial performance gain over previous state-of-the-art methods.
Previous methods for audio-image matching generally fall into one of two categories: pipeline models or End-to-End models. Pipeline models first transcribe speech and then encode the resulting text; End-to-End models encode speech directly. Generally, pipeline models outperform end-to-end models, but the intermediate transcription necessarily discards some potentially useful non-textual information. In addition to textual information, speech can convey details such as accent, mood, and and emphasis, which should be effectively captured in the encoded representation. In this paper, we investigate whether non-textual information, which is overlooked by pipeline-based models, can be leveraged to improve speech-image matching performance. We thoroughly analyze and compare End-to-End models, pipeline models, and our proposed dual-channel model for robust audio-image retrieval on a variety of datasets. Our approach achieves a substantial performance gain over the previous state-of-the-art by leveraging strong pretrained models, a prompting mechanism and a bifurcated design.