Online Aggregation of Probability Forecasts with Confidence
This work addresses the need for robust online aggregation of probability forecasts in domains like electricity consumption prediction, where expert competence varies, but it is incremental as it extends existing prediction with expert advice frameworks.
The paper tackles the problem of combining probabilistic forecasts from multiple experts in an online setting, showing that the Continuous Ranked Probability Score (CRPS) is a mixable loss function and deriving a time-independent regret bound for the Vovk aggregating algorithm.
The paper presents numerical experiments and some theoretical developments in prediction with expert advice (PEA). One experiment deals with predicting electricity consumption depending on temperature and uses real data. As the pattern of dependence can change with season and time of the day, the domain naturally admits PEA formulation with experts having different ``areas of expertise''. We consider the case where several competing methods produce online predictions in the form of probability distribution functions. The dissimilarity between a probability forecast and an outcome is measured by a loss function (scoring rule). A popular example of scoring rule for continuous outcomes is Continuous Ranked Probability Score (CRPS). In this paper the problem of combining probabilistic forecasts is considered in the PEA framework. We show that CRPS is a mixable loss function and then the time-independent upper bound for the regret of the Vovk aggregating algorithm using CRPS as a loss function can be obtained. Also, we incorporate a ``smooth'' version of the method of specialized experts in this scheme which allows us to combine the probabilistic predictions of the specialized experts with overlapping domains of their competence.