Robustness Quantification for Discriminative Models: a New Robustness Metric and its Application to Dynamic Classifier Selection
This work addresses the problem of evaluating prediction reliability for machine learning practitioners, but it is incremental as it builds on existing robustness quantification methods.
The authors tackled the limited applicability of robustness quantification for classifiers by proposing a new metric that works with any probabilistic discriminative classifier and feature type, and demonstrated its ability to distinguish reliable from unreliable predictions to develop new dynamic classifier selection strategies.
Among the different possible strategies for evaluating the reliability of individual predictions of classifiers, robustness quantification stands out as a method that evaluates how much uncertainty a classifier could cope with before changing its prediction. However, its applicability is more limited than some of its alternatives, since it requires the use of generative models and restricts the analyses either to specific model architectures or discrete features. In this work, we propose a new robustness metric applicable to any probabilistic discriminative classifier and any type of features. We demonstrate that this new metric is capable of distinguishing between reliable and unreliable predictions, and use this observation to develop new strategies for dynamic classifier selection.