Threshold Martingales and the Evolution of Forecasts
This work addresses forecast reliability for statisticians and practitioners, offering a post-processing method to enhance model outputs, though it is incremental as it builds on existing calibration adjustments.
The paper tackles the problem of evaluating and improving forecast distributions by introducing a threshold martingale that measures calibration and volatility, and proposes a martingale filter to rectify issues without modifying the original model, achieving improvements in mean squared error in simulated and basketball game prediction scenarios.
This paper introduces a martingale that characterizes two properties of evolving forecast distributions. Ideal forecasts of a future event behave as martingales, sequen- tially updating the forecast to leverage the available information as the future event approaches. The threshold martingale introduced here measures the proportion of the forecast distribution lying below a threshold. In addition to being calibrated, a threshold martingale has quadratic variation that accumulates to a total determined by a quantile of the initial forecast distribution. Deviations from calibration or to- tal volatility signal problems in the underlying model. Calibration adjustments are well-known, and we augment these by introducing a martingale filter that improves volatility while guaranteeing smaller mean squared error. Thus, post-processing can rectify problems with calibration and volatility without revisiting the original forecast- ing model. We apply threshold martingales first to forecasts from simulated models and then to models that predict the winner in professional basketball games.