Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mapping
This work addresses the challenge of soundscape mapping for applications like environmental monitoring and immersive experiences, though it is incremental as it builds on existing multimodal methods.
The authors tackled the problem of predicting sound distributions at any location by developing Sat2Sound, a multimodal framework that uses a shared codebook of soundscape concepts and contrastive learning across audio, captions, and satellite images, achieving state-of-the-art performance in cross-modal retrieval on GeoSound and SoundingEarth datasets.
We present Sat2Sound, a multimodal representation learning framework for soundscape mapping, designed to predict the distribution of sounds at any location on Earth. Existing methods for this task rely on satellite image and paired geotagged audio samples, which often fail to capture the diversity of sound sources at a given location. To address this limitation, we enhance existing datasets by leveraging a Vision-Language Model (VLM) to generate semantically rich soundscape descriptions for locations depicted in satellite images. Our approach incorporates contrastive learning across audio, audio captions, satellite images, and satellite image captions. We hypothesize that there is a fixed set of soundscape concepts shared across modalities. To this end, we learn a shared codebook of soundscape concepts and represent each sample as a weighted average of these concepts. Sat2Sound achieves state-of-the-art performance in cross-modal retrieval between satellite image and audio on two datasets: GeoSound and SoundingEarth. Additionally, building on Sat2Sound's ability to retrieve detailed soundscape captions, we introduce a novel application: location-based soundscape synthesis, which enables immersive acoustic experiences. Our code and models will be publicly available.