Evaluating the Predictive Performance of Positive-Unlabelled Classifiers: a brief critical review and practical recommendations for improvement
This addresses a methodological gap for researchers in PU learning, but it is incremental as it reviews existing practices rather than introducing new methods.
The paper tackles the problem of evaluating Positive-Unlabelled (PU) classifiers, where standard metrics are inapplicable due to missing labels, by critically reviewing evaluation approaches in 51 articles and providing practical recommendations for improvement.
Positive-Unlabelled (PU) learning is a growing area of machine learning that aims to learn classifiers from data consisting of labelled positive and unlabelled instances. Whilst much work has been done proposing methods for PU learning, little has been written on the subject of evaluating these methods. Many popular standard classification metrics cannot be precisely calculated due to the absence of fully labelled data, so alternative approaches must be taken. This short commentary paper critically reviews the main PU learning evaluation approaches and the choice of predictive accuracy measures in 51 articles proposing PU classifiers and provides practical recommendations for improvements in this area.