Bayesian Quantification with Black-Box Estimators
This work addresses the challenge of class distribution estimation for practitioners using probabilistic classifiers, though it appears incremental as it builds on existing black-box estimators.
The paper tackles the problem of estimating class distributions in unlabeled datasets for classifier calibration and uncertainty quantification by showing that existing black-box estimation algorithms are related to inference in a Bayesian model. The result demonstrates that the introduced Bayesian model is competitive and sometimes superior to state-of-the-art point estimators in various scenarios.
Understanding how different classes are distributed in an unlabeled data set is an important challenge for the calibration of probabilistic classifiers and uncertainty quantification. Approaches like adjusted classify and count, black-box shift estimators, and invariant ratio estimators use an auxiliary (and potentially biased) black-box classifier trained on a different (shifted) data set to estimate the class distribution and yield asymptotic guarantees under weak assumptions. We demonstrate that all these algorithms are closely related to the inference in a particular Bayesian model, approximating the assumed ground-truth generative process. Then, we discuss an efficient Markov Chain Monte Carlo sampling scheme for the introduced model and show an asymptotic consistency guarantee in the large-data limit. We compare the introduced model against the established point estimators in a variety of scenarios, and show it is competitive, and in some cases superior, with the state of the art.