Binary Classifier Calibration: Bayesian Non-Parametric Approach
This work addresses the need for well-calibrated predictions in decision analysis, applicable to a wide range of machine learning models, though it is incremental as it builds on existing calibration techniques.
The paper tackles the problem of calibrating probabilistic predictions from binary classifiers by introducing two new non-parametric methods based on Bayesian approaches, which are tested on various datasets and shown to either outperform or match state-of-the-art calibration methods in performance.
A set of probabilistic predictions is well calibrated if the events that are predicted to occur with probability p do in fact occur about p fraction of the time. Well calibrated predictions are particularly important when machine learning models are used in decision analysis. This paper presents two new non-parametric methods for calibrating outputs of binary classification models: a method based on the Bayes optimal selection and a method based on the Bayesian model averaging. The advantage of these methods is that they are independent of the algorithm used to learn a predictive model, and they can be applied in a post-processing step, after the model is learned. This makes them applicable to a wide variety of machine learning models and methods. These calibration methods, as well as other methods, are tested on a variety of datasets in terms of both discrimination and calibration performance. The results show the methods either outperform or are comparable in performance to the state-of-the-art calibration methods.