Faster Recalibration of an Online Predictor via Approachability
This work addresses the challenge of reliable and trustworthy predictive models in adversarial online environments, offering an incremental improvement over prior techniques.
The paper tackles the problem of ensuring calibrated probability predictions in online settings with adversarial outcomes, introducing a technique that transforms uncalibrated predictions into calibrated ones with minimal loss increase. The proposed algorithm achieves faster calibration and accuracy rates than existing methods and provides the first flexible tradeoff between calibration error and accuracy in online prediction.
Predictive models in ML need to be trustworthy and reliable, which often at the very least means outputting calibrated probabilities. This can be particularly difficult to guarantee in the online prediction setting when the outcome sequence can be generated adversarially. In this paper we introduce a technique using Blackwell's approachability theorem for taking an online predictive model which might not be calibrated and transforming its predictions to calibrated predictions without much increase to the loss of the original model. Our proposed algorithm achieves calibration and accuracy at a faster rate than existing techniques arXiv:1607.03594 and is the first algorithm to offer a flexible tradeoff between calibration error and accuracy in the online setting. We demonstrate this by characterizing the space of jointly achievable calibration and regret using our technique.