Bad and good errors: value-weighted skill scores in deep ensemble learning
This work addresses forecast verification for domains like environmental monitoring and finance, but it is incremental as it builds on existing ensemble learning methods with a new error weighting scheme.
The paper tackles the problem of assessing forecast errors by introducing a novel approach that weights errors based on the value of predictions rather than their quality, and applies this to deep ensemble learning for binary classification across pollution, space weather, and stock price forecasting applications.
In this paper we propose a novel approach to realize forecast verification. Specifically, we introduce a strategy for assessing the severity of forecast errors based on the evidence that, on the one hand, a false alarm just anticipating an occurring event is better than one in the middle of consecutive non-occurring events, and that, on the other hand, a miss of an isolated event has a worse impact than a miss of a single event, which is part of several consecutive occurrences. Relying on this idea, we introduce a novel definition of confusion matrix and skill scores giving greater importance to the value of the prediction rather than to its quality. Then, we introduce a deep ensemble learning procedure for binary classification, in which the probabilistic outcomes of a neural network are clustered via optimization of these value-weighted skill scores. We finally show the performances of this approach in the case of three applications concerned with pollution, space weather and stock prize forecasting.