WeatherReal: A Benchmark Based on In-Situ Observations for Evaluating Weather Models
This provides a more application-focused benchmark for AI weather forecasting research, addressing a domain-specific gap in evaluation methods.
The authors tackled the problem that AI-based weather models are often trained on reanalysis data that diverges from real observations, by introducing WeatherReal, a benchmark dataset based on global near-surface in-situ observations, which they used to evaluate data-driven models against numerical models.
In recent years, AI-based weather forecasting models have matched or even outperformed numerical weather prediction systems. However, most of these models have been trained and evaluated on reanalysis datasets like ERA5. These datasets, being products of numerical models, often diverge substantially from actual observations in some crucial variables like near-surface temperature, wind, precipitation and clouds - parameters that hold significant public interest. To address this divergence, we introduce WeatherReal, a novel benchmark dataset for weather forecasting, derived from global near-surface in-situ observations. WeatherReal also features a publicly accessible quality control and evaluation framework. This paper details the sources and processing methodologies underlying the dataset, and further illustrates the advantage of in-situ observations in capturing hyper-local and extreme weather through comparative analyses and case studies. Using WeatherReal, we evaluated several data-driven models and compared them with leading numerical models. Our work aims to advance the AI-based weather forecasting research towards a more application-focused and operation-ready approach.