Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error
This addresses the issue of unreliable error anticipation in machine learning models for practitioners dealing with out-of-distribution data, but it is incremental as it builds on existing coverage metrics.
The paper tackled the problem of dataset-dependence in combinatorial coverage metrics for anticipating classification error on out-of-distribution data, and found that metric learning increased the difference in set-difference coverage metrics between correctly and incorrectly classified data, validated by paired t-tests on 6 datasets.
Machine learning models are increasingly used in practice. However, many machine learning methods are sensitive to test or operational data that is dissimilar to training data. Out-of-distribution (OOD) data is known to increase the probability of error and research into metrics that identify what dissimilarities in data affect model performance is on-going. Recently, combinatorial coverage metrics have been explored in the literature as an alternative to distribution-based metrics. Results show that coverage metrics can correlate with classification error. However, other results show that the utility of coverage metrics is highly dataset-dependent. In this paper, we show that this dataset-dependence can be alleviated with metric learning, a machine learning technique for learning latent spaces where data from different classes is further apart. In a study of 6 open-source datasets, we find that metric learning increased the difference between set-difference coverage metrics (SDCCMs) calculated on correctly and incorrectly classified data, thereby demonstrating that metric learning improves the ability of SDCCMs to anticipate classification error. Paired t-tests validate the statistical significance of our findings. Overall, we conclude that metric learning improves the ability of coverage metrics to anticipate classifier error and identify when OOD data is likely to degrade model performance.