Semantic Parsing of Colonoscopy Videos with Multi-Label Temporal Networks
This work addresses the need for advanced automation tools in colonoscopy, such as quality metrics and report generation, by providing a method to parse videos into activities, events, and landmarks, though it appears incremental as it builds on existing polyp detection efforts.
The authors tackled the problem of automatically understanding colonoscopy procedure flow by developing a method for semantic parsing of colonoscopy videos, achieving evaluation on over 300 annotated videos with ablation studies to assess component importance.
Following the successful debut of polyp detection and characterization, more advanced automation tools are being developed for colonoscopy. The new automation tasks, such as quality metrics or report generation, require understanding of the procedure flow that includes activities, events, anatomical landmarks, etc. In this work we present a method for automatic semantic parsing of colonoscopy videos. The method uses a novel DL multi-label temporal segmentation model trained in supervised and unsupervised regimes. We evaluate the accuracy of the method on a test set of over 300 annotated colonoscopy videos, and use ablation to explore the relative importance of various method's components.