GEOBIND: Binding Text, Image, and Audio through Satellite Images
This work addresses the challenge of multi-modal reasoning in remote sensing for researchers and practitioners, offering a general framework that is incremental by extending existing contrastive alignment methods to satellite imagery.
The authors tackled the problem of modeling multiple modalities (text, image, audio) from satellite imagery by introducing GeoBind, a deep-learning model that uses satellite images as a binding element to align these modalities contrastively, resulting in a versatile joint embedding space without requiring a complex multi-modal dataset.
In remote sensing, we are interested in modeling various modalities for some geographic location. Several works have focused on learning the relationship between a location and type of landscape, habitability, audio, textual descriptions, etc. Recently, a common way to approach these problems is to train a deep-learning model that uses satellite images to infer some unique characteristics of the location. In this work, we present a deep-learning model, GeoBind, that can infer about multiple modalities, specifically text, image, and audio, from satellite imagery of a location. To do this, we use satellite images as the binding element and contrastively align all other modalities to the satellite image data. Our training results in a joint embedding space with multiple types of data: satellite image, ground-level image, audio, and text. Furthermore, our approach does not require a single complex dataset that contains all the modalities mentioned above. Rather it only requires multiple satellite-image paired data. While we only align three modalities in this paper, we present a general framework that can be used to create an embedding space with any number of modalities by using satellite images as the binding element. Our results show that, unlike traditional unimodal models, GeoBind is versatile and can reason about multiple modalities for a given satellite image input.