SANGRIA: Surgical Video Scene Graph Optimization for Surgical Workflow Prediction
This work addresses the costly annotation burden for semantic scene comprehension in surgical videos, benefiting medical AI applications, but it is incremental as it builds on existing graph-based methods.
The paper tackles the problem of limited densely annotated surgical scene data by introducing an end-to-end framework for generating and optimizing surgical scene graphs, which improves surgical workflow prediction by achieving an 8% accuracy and 10% F1 score gain over state-of-the-art methods on the CATARACTS dataset.
Graph-based holistic scene representations facilitate surgical workflow understanding and have recently demonstrated significant success. However, this task is often hindered by the limited availability of densely annotated surgical scene data. In this work, we introduce an end-to-end framework for the generation and optimization of surgical scene graphs on a downstream task. Our approach leverages the flexibility of graph-based spectral clustering and the generalization capability of foundation models to generate unsupervised scene graphs with learnable properties. We reinforce the initial spatial graph with sparse temporal connections using local matches between consecutive frames to predict temporally consistent clusters across a temporal neighborhood. By jointly optimizing the spatiotemporal relations and node features of the dynamic scene graph with the downstream task of phase segmentation, we address the costly and annotation-burdensome task of semantic scene comprehension and scene graph generation in surgical videos using only weak surgical phase labels. Further, by incorporating effective intermediate scene representation disentanglement steps within the pipeline, our solution outperforms the SOTA on the CATARACTS dataset by 8% accuracy and 10% F1 score in surgical workflow recognition