A Clinician-Centered Pipeline for Annotation and Evaluation in Ultrasound AI Studies
For clinicians and researchers in ultrasound AI, this pipeline addresses the lack of integrated tools for blinded model comparison and reproducible evaluation, but the contribution is incremental as it combines existing components.
The paper presents a clinician-centered pipeline for remote annotation and blinded evaluation of ultrasound AI models, validated in a fetal ultrasound segmentation study with six raters. The pipeline enabled automated statistical analysis and showed moderate to strong inter-rater agreement, with a tendency for later active learning models to be preferred.
Clinician-centered evaluation is critical for validating medical AI systems, especially in ultrasound imaging where quantitative metrics do not always capture clinical usability. Existing medical image platforms primarily focus on dataset labeling. They lack integrated support for blinded model comparison and reproducible evaluation workflows. We present a clinician-centered pipeline for remote annotation and evaluation in ultrasound AI studies. The proposed pipeline uses a centralized server and lightweight browser interfaces to enable clinicians to perform annotation, blinded ranking, and review without local dataset downloads. The pipeline also supports multi-rater participation, centralized result aggregation, and automated statistical analysis. We validate the pipeline in a fetal ultrasound segmentation study with six raters spanning expert, generalist, and non-expert experience levels. The system automatically generated Spearman correlation, Kendall's $τ$, and top-1 selection statistics. Results indicated moderate to strong agreement across experts and other groups. The blinded evaluation results showed a tendency for later active learning models to be preferred. These outcomes suggest that the pipeline can support clinician-centered annotation and reproducible human-\ac{AI} evaluation studies in ultrasound imaging. The proposed pipeline is available on \href{https://github.com/13204942/SonoRate}{GitHub}.