Assessing binary classifiers using only positive and unlabeled data
This addresses a practical limitation in domains like medical diagnosis or fraud detection where labeled negative examples are scarce, offering a solution for model assessment without full labels.
The paper tackles the problem of evaluating binary classifiers when only positive and unlabeled data are available, proposing a method to estimate metrics like ROC and PR curves by estimating the fraction of positives in the unlabeled set, with empirical validation on real data showing reliable estimates.
Assessing the performance of a learned model is a crucial part of machine learning. However, in some domains only positive and unlabeled examples are available, which prohibits the use of most standard evaluation metrics. We propose an approach to estimate any metric based on contingency tables, including ROC and PR curves, using only positive and unlabeled data. Estimating these performance metrics is essentially reduced to estimating the fraction of (latent) positives in the unlabeled set, assuming known positives are a random sample of all positives. We provide theoretical bounds on the quality of our estimates, illustrate the importance of estimating the fraction of positives in the unlabeled set and demonstrate empirically that we are able to reliably estimate ROC and PR curves on real data.