Honest and Reliable Evaluation and Expert Equivalence Testing of Automated Neonatal Seizure Detection
This addresses the critical need for standardized evaluation in neonatal seizure detection AI to enable clinical adoption, though it is incremental as it focuses on improving existing evaluation practices rather than developing new detection methods.
This study tackled the problem of unreliable evaluation metrics for neonatal seizure detection AI models by systematically assessing performance metrics and proposing best practices, finding that Matthews and Pearson's correlation coefficients outperformed AUC under class imbalance and recommending a multi-rater Turing test using Fleiss k for expert-level validation.
Reliable evaluation of machine learning models for neonatal seizure detection is critical for clinical adoption. Current practices often rely on inconsistent and biased metrics, hindering model comparability and interpretability. Expert-level claims about AI performance are frequently made without rigorous validation, raising concerns about their reliability. This study aims to systematically evaluate common performance metrics and propose best practices tailored to the specific challenges of neonatal seizure detection. Using real and synthetic seizure annotations, we assessed standard performance metrics, consensus strategies, and human-expert level equivalence tests under varying class imbalance, inter-rater agreement, and number of raters. Matthews and Pearson's correlation coefficients outperformed the area under the curve in reflecting performance under class imbalance. Consensus types are sensitive to the number of raters and agreement level among them. Among human-expert level equivalence tests, the multi-rater Turing test using Fleiss k best captured expert-level AI performance. We recommend reporting: (1) at least one balanced metric, (2) Sensitivity, specificity, PPV and NPV, (3) Multi-rater Turing test results using Fleiss k, and (4) All the above on held-out validation set. This proposed framework provides an important prerequisite to clinical validation by enabling a thorough and honest appraisal of AI methods for neonatal seizure detection.