No One Knows the State of the Art in Geospatial Foundation Models
For researchers and practitioners in geospatial AI, this paper identifies a critical lack of reproducibility and comparability in GFM research, proposing concrete standards to address it.
The paper argues that the geospatial foundation model (GFM) literature lacks standardized evaluations, protocols, and weight releases, making it impossible to determine the state of the art. An audit of 152 papers found 46 cross-paper disagreements of at least 10 points on the same model and benchmark, 94/126 papers with unique pretraining configurations, and 39% releasing no weights.
Geospatial foundation models (GFMs) have been proposed as generalizable backbones for disaster response, land-cover mapping, food-security monitoring, and other high-stakes Earth-observation tasks. Yet the published work about these models does not give reviewers or users enough information to tell which model fits a given task. We argue that nobody knows what the current state of the art is in geospatial foundation models. The methods may be useful, but the GFM literature does not standardize evaluations, training and testing protocols, released weights, or pretraining controls well enough for anyone to compare or rank them. In a 152-paper audit, we find 46 cross-paper disagreements of at least 10 points for the same model, benchmark, and protocol; 94/126 papers with extractable pretraining data use a configuration no other paper uses; and 39% of GFM papers release no model weights. This lack of community standards can be solved. We propose six concrete expectations: named-license weight release, shared core evaluations, copied-versus-rerun baseline annotations, variance reporting, one shared evaluation harness, and data-vs-architecture-vs-algorithm controls. These gaps are a coordination failure, not a fault of any individual lab; the authors of this paper, like many others in the GFM community, have contributed to them. Rather than just critiquing the community, we aim to provide concrete steps toward a shared understanding of how to innovate GFMs.