Human and AI collaboration for pulmonary nodule segmentation
For medical image segmentation, Hi-Seg demonstrates that human-AI collaboration can reduce clinician workload and enable scalable crowdsourced annotation, though the approach is incremental as it builds on existing SAM technology.
Hi-Seg, a human-in-the-loop segmentation framework built on SAM, achieved a mean Dice score of ~85% for pulmonary nodule segmentation, outperforming five state-of-the-art deep learning models by 10-22% and 13 SAM variants by 1-29%, while reducing annotation time and enabling non-medical annotators to match junior medical students.
Medical expert annotators are scarce, and blind reliance on artificial intelligence (AI) can be misleading, motivating approaches in which humans, particularly junior medical trainees or even non-medical personnel, collaborate with AI to achieve robust medical segmentation. Although the Segment Anything Model (SAM) shows promise for general-purpose image segmentation, its performance in human-AI collaboration for specialized medical tasks has not been thoroughly evaluated. Here we present Hi-Seg, a human-in-the-loop segmentation framework for pulmonary nodules built on SAM. Humans iteratively refine prompts through trial-and-error learning and semantic reasoning, progressively guiding SAM toward higher-quality masks. Using chest CT scans from 1,179 patients across 12 centers, we conducted the first large-scale external validation of collaborative human-SAM segmentation. Across all annotator groups, Hi-Seg achieved a mean Dice score of almost 85%, outperforming five state-of-the-art deep learning models by 10-22% and 13 SAM variants by 1-29%. Hi-Seg improved segmentation accuracy while reducing annotation time for medical annotators, and briefly trained non-medical annotators achieved performance comparable to that of the junior medical student. These findings suggest that human-in-the-loop segmentation can reduce clinician workload, enable scalable crowdsourced annotation, and transform clinical workflows by facilitating the safe and efficient integration of foundation models into routine clinical practice.