Efficient, Anytime Algorithms for Calibration with Isotonic Regression under Strictly Convex Losses
This work addresses calibration in machine learning for improved estimator performance, offering efficient algorithms but is incremental as it builds on existing isotonic regression methods.
The paper tackles the problem of calibrating estimations using optimal monotone transforms under strictly convex losses, showing that the optimal transform is a unique staircase function and proposing linear-time algorithms for specific loss settings, including an online version with linear space and amortized time.
We investigate the calibration of estimations to increase performance with an optimal monotone transform on the estimator outputs. We start by studying the traditional square error setting with its weighted variant and show that the optimal monotone transform is in the form of a unique staircase function. We further show that this staircase behavior is preserved for general strictly convex loss functions. Their optimal monotone transforms are also unique, i.e., there exist a single staircase transform that achieves the minimum loss. We propose a linear time and space algorithm that can find such optimal transforms for specific loss settings. Our algorithm has an online implementation where the optimal transform for the samples observed so far are found in linear space and amortized time when the samples arrive in an ordered fashion. We also extend our results to cases where the functions are not trivial to individually optimize and propose an anytime algorithm, which has linear space and pseudo-linearithmic time complexity.