Estimating the Accuracies of Multiple Classifiers Without Labeled Data
This work addresses a practical challenge in machine learning for scenarios where labeled data is unavailable, offering a solution for unsupervised accuracy estimation and ensemble construction, though it is incremental as it builds on standard independence assumptions.
The paper tackles the problem of estimating the accuracies of multiple classifiers and constructing an ensemble classifier without any labeled data or prior knowledge, focusing on binary classification. It presents simple, computationally efficient algorithms that are proven consistent under independence assumptions, achieving competitive performance in experiments on artificial and real datasets.
In various situations one is given only the predictions of multiple classifiers over a large unlabeled test data. This scenario raises the following questions: Without any labeled data and without any a-priori knowledge about the reliability of these different classifiers, is it possible to consistently and computationally efficiently estimate their accuracies? Furthermore, also in a completely unsupervised manner, can one construct a more accurate unsupervised ensemble classifier? In this paper, focusing on the binary case, we present simple, computationally efficient algorithms to solve these questions. Furthermore, under standard classifier independence assumptions, we prove our methods are consistent and study their asymptotic error. Our approach is spectral, based on the fact that the off-diagonal entries of the classifiers' covariance matrix and 3-d tensor are rank-one. We illustrate the competitive performance of our algorithms via extensive experiments on both artificial and real datasets.