MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video
This addresses the need for efficient clinical workflow in medical imaging, specifically for fetal ultrasound analysis, by automating a process that mimics human sonographers, though it appears incremental as it adapts existing methods like large language models for a new domain.
The authors tackled the problem of automating summary generation for fetal ultrasound videos by developing MMSummary, a three-stage multimodal system that detects keyframes, generates captions, and segments anatomy, resulting in an estimated 31.5% reduction in scanning time.
We present the first automated multimodal summary generation system, MMSummary, for medical imaging video, particularly with a focus on fetal ultrasound analysis. Imitating the examination process performed by a human sonographer, MMSummary is designed as a three-stage pipeline, progressing from keyframe detection to keyframe captioning and finally anatomy segmentation and measurement. In the keyframe detection stage, an innovative automated workflow is proposed to progressively select a concise set of keyframes, preserving sufficient video information without redundancy. Subsequently, we adapt a large language model to generate meaningful captions for fetal ultrasound keyframes in the keyframe captioning stage. If a keyframe is captioned as fetal biometry, the segmentation and measurement stage estimates biometric parameters by segmenting the region of interest according to the textual prior. The MMSummary system provides comprehensive summaries for fetal ultrasound examinations and based on reported experiments is estimated to reduce scanning time by approximately 31.5%, thereby suggesting the potential to enhance clinical workflow efficiency.