Tournament Leave-pair-out Cross-validation for Receiver Operating Characteristic (ROC) Analysis
This addresses a specific issue in diagnostic system evaluation for researchers, but it is incremental as it builds directly on existing LPO methods.
The authors tackled the problem of bias in AUC estimation and lack of ranking for ROC analysis in cross-validation by proposing tournament leave-pair-out (TLPO) cross-validation, which extends LPO to produce rankings while maintaining reliable AUC estimation, as validated on synthetic and real-world data.
Receiver operating characteristic (ROC) analysis is widely used for evaluating diagnostic systems. Recent studies have shown that estimating an area under ROC curve (AUC) with standard cross-validation methods suffers from a large bias. The leave-pair-out (LPO) cross-validation has been shown to correct this bias. However, while LPO produces an almost unbiased estimate of AUC, it does not provide a ranking of the data needed for plotting and analyzing the ROC curve. In this study, we propose a new method called tournament leave-pair-out (TLPO) cross-validation. This method extends LPO by creating a tournament from pair comparisons to produce a ranking for the data. TLPO preserves the advantage of LPO for estimating AUC, while it also allows performing ROC analyses. We have shown using both synthetic and real world data that TLPO is as reliable as LPO for AUC estimation, and confirmed the bias in leave-one-out cross-validation on low-dimensional data. As a case study on ROC analysis, we also evaluate how reliably sensitivity and specificity can be estimated from TLPO ROC curves.