Learning Tri-modal Embeddings for Zero-Shot Soundscape Mapping
This work addresses soundscape mapping for geographic analysis, but it is incremental as it builds on existing state-of-the-art models.
The paper tackles the problem of soundscape mapping by predicting probable sounds at geographic locations, achieving a significant improvement in image-to-audio Recall@100 from 0.256 to 0.450 on the SoundingEarth dataset.
We focus on the task of soundscape mapping, which involves predicting the most probable sounds that could be perceived at a particular geographic location. We utilise recent state-of-the-art models to encode geotagged audio, a textual description of the audio, and an overhead image of its capture location using contrastive pre-training. The end result is a shared embedding space for the three modalities, which enables the construction of soundscape maps for any geographic region from textual or audio queries. Using the SoundingEarth dataset, we find that our approach significantly outperforms the existing SOTA, with an improvement of image-to-audio Recall@100 from 0.256 to 0.450. Our code is available at https://github.com/mvrl/geoclap.