Signature Kernel Scoring Rule as Spatio-Temporal Diagnostic for Probabilistic Forecasting
This work addresses the need for better diagnostic tools in probabilistic weather forecasting, offering a novel scoring rule that improves model training and evaluation for weather prediction systems.
The paper tackles the problem of evaluating and training probabilistic weather forecasting models by introducing the signature kernel scoring rule, which captures spatio-temporal dependencies ignored by conventional metrics like MSE, and demonstrates that training with this rule outperforms climatology for forecasts up to fifteen timesteps using a lightweight model.
Modern weather forecasting has increasingly transitioned from numerical weather prediction (NWP) to data-driven machine learning forecasting techniques. While these new models produce probabilistic forecasts to quantify uncertainty, their training and evaluation may remain hindered by conventional scoring rules, primarily MSE, which ignore the highly correlated data structures present in weather and atmospheric systems. This work introduces the signature kernel scoring rule, grounded in rough path theory, which reframes weather variables as continuous paths to encode temporal and spatial dependencies through iterated integrals. Validated as strictly proper through the use of path augmentations to guarantee uniqueness, the signature kernel provides a theoretically robust metric for forecast verification and model training. Empirical evaluations through weather scorecards on WeatherBench 2 models demonstrate the signature kernel scoring rule's high discriminative power and unique capacity to capture path-dependent interactions. Following previous demonstration of successful adversarial-free probabilistic training, we train sliding window generative neural networks using a predictive-sequential scoring rule on ERA5 reanalysis weather data. Using a lightweight model, we demonstrate that signature kernel based training outperforms climatology for forecast paths of up to fifteen timesteps.