DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applications
This work addresses the limitation of coarse embeddings for Earth observation applications, improving integration with modalities like LiDAR for tasks such as canopy height mapping and land cover classification, though it is incremental as it builds on existing self-supervised multimodal learning methods.
The paper tackles the problem of coarse patch-sized embeddings in Earth observation by introducing DUNIA, which learns pixel-sized embeddings through cross-modal alignment between images and LiDAR data, enabling zero-shot performance that often outperforms specialized supervised models in seven environmental monitoring tasks.
Significant efforts have been directed towards adapting self-supervised multimodal learning for Earth observation applications. However, most current methods produce coarse patch-sized embeddings, limiting their effectiveness and integration with other modalities like LiDAR. To close this gap, we present DUNIA, an approach to learn pixel-sized embeddings through cross-modal alignment between images and full-waveform LiDAR data. As the model is trained in a contrastive manner, the embeddings can be directly leveraged in the context of a variety of environmental monitoring tasks in a zero-shot setting. In our experiments, we demonstrate the effectiveness of the embeddings for seven such tasks: canopy height mapping, fractional canopy cover, land cover mapping, tree species identification, plant area index, crop type classification, and per-pixel waveform-based vertical structure mapping. The results show that the embeddings, along with zero-shot classifiers, often outperform specialized supervised models, even in low-data regimes. In the fine-tuning setting, we show strong performances near or better than the state-of-the-art on five out of six tasks.