A metrological framework for uncertainty evaluation in machine learning classification models
This work addresses a foundational gap in uncertainty quantification for classification models, which is crucial for high-stakes applications like climate science and healthcare, though it is incremental in extending existing metrological standards.
The paper tackles the lack of uncertainty evaluation methods for nominal properties in machine learning classification models, proposing a metrological framework based on probability mass functions and summary statistics, and illustrates its application in climate/earth observation and medical diagnosis.
Machine learning (ML) classification models are increasingly being used in a wide range of applications where it is important that predictions are accompanied by uncertainties, including in climate and earth observation, medical diagnosis and bioaerosol monitoring. The output of an ML classification model is a type of categorical variable known as a nominal property in the International Vocabulary of Metrology (VIM). However, concepts related to uncertainty evaluation for nominal properties are not defined in the VIM, nor is such evaluation addressed by the Guide to the Expression of Uncertainty in Measurement (GUM). In this paper we propose a metrological conceptual uncertainty evaluation framework for nominal properties. This framework is based on probability mass functions and summary statistics thereof, and it is applicable to ML classification. We also illustrate its use in the context of two applications that exemplify the issues and have significant societal impact, namely, climate and earth observation and medical diagnosis. Our framework would enable an extension of the GUM to uncertainty for nominal properties, which would make both applicable to ML classification models.