Uncertainty Quantification for Rule-Based Models
This work addresses uncertainty estimation for rule-based models, a domain-specific challenge in machine learning, but is incremental as it builds on existing black-box approaches.
The authors tackled the problem of uncertainty quantification for rule-based models, which lack continuous outputs, by proposing a meta-model that estimates prediction accuracy and confidence for any binary classifier, and demonstrated its utility through an abstaining classifier.
Rule-based classification models described in the language of logic directly predict boolean values, rather than modeling a probability and translating it into a prediction as done in statistical models. The vast majority of existing uncertainty quantification approaches rely on models providing continuous output not available to rule-based models. In this work, we propose an uncertainty quantification framework in the form of a meta-model that takes any binary classifier with binary output as a black box and estimates the prediction accuracy of that base model at a given input along with a level of confidence on that estimation. The confidence is based on how well that input region is explored and is designed to work in any OOD scenario. We demonstrate the usefulness of this uncertainty model by building an abstaining classifier powered by it and observing its performance in various scenarios.