Model Comparison and Calibration Assessment: User Guide for Consistent Scoring Functions in Machine Learning and Actuarial Practice
It provides practical guidance for actuaries and data scientists to improve model assessment and comparison, but is incremental as it focuses on pedagogical presentation of existing results and best practices.
This user guide revisits and clarifies statistical techniques for assessing model calibration and comparing models in machine learning and actuarial practice, emphasizing the importance of aligning scoring functions with prediction targets, and illustrates the results with real data case studies on workers' compensation and customer churn.
One of the main tasks of actuaries and data scientists is to build good predictive models for certain phenomena such as the claim size or the number of claims in insurance. These models ideally exploit given feature information to enhance the accuracy of prediction. This user guide revisits and clarifies statistical techniques to assess the calibration or adequacy of a model on the one hand, and to compare and rank different models on the other hand. In doing so, it emphasises the importance of specifying the prediction target functional at hand a priori (e.g. the mean or a quantile) and of choosing the scoring function in model comparison in line with this target functional. Guidance for the practical choice of the scoring function is provided. Striving to bridge the gap between science and daily practice in application, it focuses mainly on the pedagogical presentation of existing results and of best practice. The results are accompanied and illustrated by two real data case studies on workers' compensation and customer churn.