PyViT-FUSE: A Foundation Model for Multi-Sensor Earth Observation Data
This addresses the challenge of handling diverse sensor data in earth observation for researchers and practitioners, though it appears incremental as it builds on existing vision transformer and self-supervised learning methods.
The authors tackled the problem of processing multi-modal earth observation data by proposing PyViT-FUSE, a foundation model that fuses arbitrary mixed-resolution input bands into a single representation using attention, and demonstrated its interpretability and applicability to downstream tasks.
We propose PyViT-FUSE, a foundation model for earth observation data explicitly designed to handle multi-modal imagery by learning to fuse an arbitrary number of mixed-resolution input bands into a single representation through an attention mechanism. The learned patch tokens are further processed by a stack of vision transformers with a novel pyramidal structure. We train the model on a globally sampled dataset in a self-supervised manner, leveraging core concepts of the SwAV algorithm. We show the interpretability of the fusion mechanism by visualization of the attention scores and the models applicability to downstream tasks.