Robustness investigation of cross-validation based quality measures for model assessment
This work addresses the reliability of model assessment metrics for researchers and practitioners, but it is incremental as it builds on existing cross-validation methods.
The paper investigates the accuracy and robustness of cross-validation-based quality measures for assessing machine learning models, quantifying explained variation and estimating confidence bounds through numerical examples with verification datasets.
In this paper the accuracy and robustness of quality measures for the assessment of machine learning models are investigated. The prediction quality of a machine learning model is evaluated model-independent based on a cross-validation approach, where the approximation error is estimated for unknown data. The presented measures quantify the amount of explained variation in the model prediction. The reliability of these measures is assessed by means of several numerical examples, where an additional data set for the verification of the estimated prediction error is available. Furthermore, the confidence bounds of the presented quality measures are estimated and local quality measures are derived from the prediction residuals obtained by the cross-validation approach.