Recovering True Classifier Performance in Positive-Unlabeled Learning
This work solves the issue of inaccurate classifier evaluation in positive-unlabeled learning, which is crucial for applications like medical diagnosis or fraud detection, though it is incremental as it builds on existing correction methods.
The paper addresses the problem of biased performance estimates in positive-unlabeled learning by showing that standard measures like ROC and precision-recall curves can be corrected using class priors and noise proportions, and demonstrates efficacy on real-life data with state-of-the-art estimation algorithms.
A common approach in positive-unlabeled learning is to train a classification model between labeled and unlabeled data. This strategy is in fact known to give an optimal classifier under mild conditions; however, it results in biased empirical estimates of the classifier performance. In this work, we show that the typically used performance measures such as the receiver operating characteristic curve, or the precision-recall curve obtained on such data can be corrected with the knowledge of class priors; i.e., the proportions of the positive and negative examples in the unlabeled data. We extend the results to a noisy setting where some of the examples labeled positive are in fact negative and show that the correction also requires the knowledge of the proportion of noisy examples in the labeled positives. Using state-of-the-art algorithms to estimate the positive class prior and the proportion of noise, we experimentally evaluate two correction approaches and demonstrate their efficacy on real-life data.