The Transparent Earth: A Multimodal Foundation Model for the Earth's Subsurface
This provides a scalable foundation model for predicting Earth's subsurface properties, addressing a domain-specific challenge in geoscience with incremental improvements in handling multimodal data.
The paper tackles the problem of reconstructing subsurface properties from heterogeneous datasets by introducing a transformer-based architecture that scales to arbitrary modalities and supports in-context learning, reducing errors in predicting stress angle by more than a factor of three on validation data.
We present the Transparent Earth, a transformer-based architecture for reconstructing subsurface properties from heterogeneous datasets that vary in sparsity, resolution, and modality, where each modality represents a distinct type of observation (e.g., stress angle, mantle temperature, tectonic plate type). The model incorporates positional encodings of observations together with modality encodings, derived from a text embedding model applied to a description of each modality. This design enables the model to scale to an arbitrary number of modalities, making it straightforward to add new ones not considered in the initial design. We currently include eight modalities spanning directional angles, categorical classes, and continuous properties such as temperature and thickness. These capabilities support in-context learning, enabling the model to generate predictions either with no inputs or with an arbitrary number of additional observations from any subset of modalities. On validation data, this reduces errors in predicting stress angle by more than a factor of three. The proposed architecture is scalable and demonstrates improved performance with increased parameters. Together, these advances make the Transparent Earth an initial foundation model for the Earth's subsurface that ultimately aims to predict any subsurface property anywhere on Earth.