Calibration and Discrimination Optimization Using Clusters of Learned Representation
This addresses the need for better calibration in critical applications like clinical predictions, though it appears incremental as it builds on existing calibration methods.
The paper tackles the problem of improving calibration in machine learning models for reliable decision-making, introducing a calibration pipeline that uses clusters of learned representations to enhance calibration scores from 82.28% up to 100% while optimizing both discrimination and calibration.
Machine learning models are essential for decision-making and risk assessment, requiring highly reliable predictions in terms of both discrimination and calibration. While calibration often receives less attention, it is crucial for critical decisions, such as those in clinical predictions. We introduce a novel calibration pipeline that leverages an ensemble of calibration functions trained on clusters of learned representations of the input samples to enhance overall calibration. This approach not only improves the calibration score of various methods from 82.28% up to 100% but also introduces a unique matching metric that ensures model selection optimizes both discrimination and calibration. Our generic scheme adapts to any underlying representation, clustering, calibration methods and metric, offering flexibility and superior performance across commonly used calibration methods.