Confusion Matrices and Accuracy Statistics for Binary Classifiers Using Unlabeled Data: The Diagnostic Test Approach
This work addresses a practical challenge in machine learning for researchers and practitioners needing to evaluate classifiers in scenarios with limited labeled data, though it is incremental as it adapts existing medical methods.
The paper tackles the problem of estimating confusion matrices and accuracy statistics for binary classifiers when only unlabeled data is available, by adapting methods from medical diagnostic test literature to achieve this without requiring a gold standard.
Medical researchers have solved the problem of estimating the sensitivity and specificity of binary medical diagnostic tests without gold standard tests for comparison. That problem is the same as estimating confusion matrices for classifiers on unlabeled data. This article describes how to modify the diagnostic test solutions to estimate confusion matrices and accuracy statistics for supervised or unsupervised binary classifiers on unlabeled data.