Confidence Matters: Uncertainty Quantification and Precision Assessment of Deep Learning-based CMR Biomarker Estimates Using Scan-rescan Data
For researchers and clinicians using deep learning for cardiac MRI analysis, this paper highlights the inadequacy of standard accuracy metrics and the need for precision-aware evaluation, but the findings are incremental as they confirm known limitations of point estimates.
This work applies uncertainty estimation techniques to a deep learning pipeline for cardiac MRI biomarker estimation and proposes new distribution-based metrics for precision assessment. While point estimate accuracy was high (average Dice 87%), distributional analyses showed poor scan-rescan agreement, with overlap >50% in less than 45% of cases and significant differences in over 65% of cases.
The performance of deep learning (DL) methods for the analysis of cine cardiovascular magnetic resonance (CMR) is typically assessed in terms of accuracy, overlooking precision. In this work, uncertainty estimation techniques, namely deep ensemble, test-time augmentation, and Monte Carlo dropout, are applied to a state-of-the-art DL pipeline for cardiac functional biomarker estimation, and new distribution-based metrics are proposed for the assessment of biomarker precision. The model achieved high accuracy (average Dice 87%) and point estimate precision on two external validation scan-rescan CMR datasets. However, distribution-based metrics showed that the overlap between scan/rescan confidence intervals was >50% in less than 45% of the cases. Statistical similarity tests between scan and rescan biomarkers also resulted in significant differences for over 65% of the cases. We conclude that, while point estimate metrics might suggest good performance, distributional analyses reveal lower precision, highlighting the need to use more representative metrics to assess scan-rescan agreement.