Online Platt Scaling with Calibeating
This addresses calibration issues in online settings for practitioners dealing with distribution shifts, though it is incremental as it builds on existing Platt scaling and calibeating techniques.
The paper tackles the problem of online post-hoc calibration for machine learning models by introducing Online Platt Scaling (OPS), which adapts to i.i.d. and non-i.i.d. settings with distribution drift, and enhances it with calibeating for robustness against adversarial outcomes, achieving superior performance without hyperparameter tuning on synthetic and real-world datasets.
We present an online post-hoc calibration method, called Online Platt Scaling (OPS), which combines the Platt scaling technique with online logistic regression. We demonstrate that OPS smoothly adapts between i.i.d. and non-i.i.d. settings with distribution drift. Further, in scenarios where the best Platt scaling model is itself miscalibrated, we enhance OPS by incorporating a recently developed technique called calibeating to make it more robust. Theoretically, our resulting OPS+calibeating method is guaranteed to be calibrated for adversarial outcome sequences. Empirically, it is effective on a range of synthetic and real-world datasets, with and without distribution drifts, achieving superior performance without hyperparameter tuning. Finally, we extend all OPS ideas to the beta scaling method.