Classifier Risk Estimation under Limited Labeling Resources
This addresses the challenge of limited labeling resources for practitioners in machine learning, offering an incremental improvement over existing sampling methods.
The paper tackles the problem of estimating classifier performance when only a small subset of test data can be labeled, proposing stratified sampling strategies to select subsets that yield accurate estimates. The results show a variance reduction of over 65% compared to random sampling and up to 60% fewer samples needed for a 1% error.
In this paper we propose strategies for estimating performance of a classifier when labels cannot be obtained for the whole test set. The number of test instances which can be labeled is very small compared to the whole test data size. The goal then is to obtain a precise estimate of classifier performance using as little labeling resource as possible. Specifically, we try to answer, how to select a subset of the large test set for labeling such that the performance of a classifier estimated on this subset is as close as possible to the one on the whole test set. We propose strategies based on stratified sampling for selecting this subset. We show that these strategies can reduce the variance in estimation of classifier accuracy by a significant amount compared to simple random sampling (over 65% in several cases). Hence, our proposed methods are much more precise compared to random sampling for accuracy estimation under restricted labeling resources. The reduction in number of samples required (compared to random sampling) to estimate the classifier accuracy with only 1% error is high as 60% in some cases.