Unsupervised Recalibration
This addresses the issue of model performance degradation in real-world deployments for users needing reliable predictions without retraining or labeled data, though it is an incremental improvement over existing calibration methods.
The paper tackles the problem of improving the accuracy of trained probabilistic models when deployed on new, unlabeled field data by introducing Unsupervised Recalibration (URC), which corrects bias without requiring ground truth and can handle subpopulations not considered during training.
Unsupervised recalibration (URC) is a general way to improve the accuracy of an already trained probabilistic classification or regression model upon encountering new data while deployed in the field. URC does not require any ground truth associated with the new field data. URC merely observes the model's predictions and recognizes when the training set is not representative of field data, and then corrects to remove any introduced bias. URC can be particularly useful when applied separately to different subpopulations observed in the field that were not considered as features when training the machine learning model. This makes it possible to exploit subpopulation information without retraining the model or even having ground truth for some or all subpopulations available. Additionally, if these subpopulations are the object of study, URC serves to determine the correct ground truth distributions for them, where naive aggregation methods, like averaging the model's predictions, systematically underestimate their differences.