Fixing the Pitfalls of Probabilistic Time-Series Forecasting Evaluation by Kernel Quadrature
This addresses a critical evaluation issue for researchers and practitioners in time-series forecasting, though it is incremental as it improves upon existing methods rather than introducing a new paradigm.
The paper tackled the problem of biased estimators for the continuous ranked probability score (CRPS) in probabilistic time-series forecasting, which cause improper model rankings, and introduced a kernel quadrature approach that consistently outperforms existing estimators.
Despite the significance of probabilistic time-series forecasting models, their evaluation metrics often involve intractable integrations. The most widely used metric, the continuous ranked probability score (CRPS), is a strictly proper scoring function; however, its computation requires approximation. We found that popular CRPS estimators--specifically, the quantile-based estimator implemented in the widely used GluonTS library and the probability-weighted moment approximation--both exhibit inherent estimation biases. These biases lead to crude approximations, resulting in improper rankings of forecasting model performance when CRPS values are close. To address this issue, we introduced a kernel quadrature approach that leverages an unbiased CRPS estimator and employs cubature construction for scalable computation. Empirically, our approach consistently outperforms the two widely used CRPS estimators.