Procedure-Aware Surgical Video-language Pretraining with Hierarchical Knowledge Augmentation
This work provides a generalist visual representation for surgical scene understanding, benefiting researchers and practitioners in medical AI by improving zero-shot transfer capabilities, though the specific problem for whom is not explicitly quantified.
This study addresses challenges in surgical video-language pretraining by proposing a hierarchical knowledge augmentation approach and a novel framework called PeskaVLP. It uses large language models to refine surgical concepts and combines language supervision with visual self-supervision, achieving significant improvements in zero-shot transfer performance on multiple surgical scene understanding and cross-modal retrieval datasets.
Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge domain gap and the scarcity of multi-modal data. This study aims to bridge the gap by addressing issues regarding textual information loss in surgical lecture videos and the spatial-temporal challenges of surgical VLP. We propose a hierarchical knowledge augmentation approach and a novel Procedure-Encoded Surgical Knowledge-Augmented Video-Language Pretraining (PeskaVLP) framework to tackle these issues. The knowledge augmentation uses large language models (LLM) for refining and enriching surgical concepts, thus providing comprehensive language supervision and reducing the risk of overfitting. PeskaVLP combines language supervision with visual self-supervision, constructing hard negative samples and employing a Dynamic Time Warping (DTW) based loss function to effectively comprehend the cross-modal procedural alignment. Extensive experiments on multiple public surgical scene understanding and cross-modal retrieval datasets show that our proposed method significantly improves zero-shot transferring performance and offers a generalist visual representation for further advancements in surgical scene understanding.The code is available at https://github.com/CAMMA-public/SurgVLP