Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery
This work addresses the need for automated event prediction to augment surgeons' perception and decision-making during laparoscopic and robotic surgery, representing an incremental improvement with domain-specific applications.
The paper tackles the problem of predicting intraoperative events and actions in minimally invasive surgery by proposing a hypergraph-transformer (HGT) neural network that leverages surgical knowledge graphs, achieving superior results compared to unstructured alternatives on tasks like action triplet prediction and Critical View of Safety (CVS) achievement.
Understanding and anticipating intraoperative events and actions is critical for intraoperative assistance and decision-making during minimally invasive surgery. Automated prediction of events, actions, and the following consequences is addressed through various computational approaches with the objective of augmenting surgeons' perception and decision-making capabilities. We propose a predictive neural network that is capable of understanding and predicting critical interactive aspects of surgical workflow from intra-abdominal video, while flexibly leveraging surgical knowledge graphs. The approach incorporates a hypergraph-transformer (HGT) structure that encodes expert knowledge into the network design and predicts the hidden embedding of the graph. We verify our approach on established surgical datasets and applications, including the detection and prediction of action triplets, and the achievement of the Critical View of Safety (CVS). Moreover, we address specific, safety-related tasks, such as predicting the clipping of cystic duct or artery without prior achievement of the CVS. Our results demonstrate the superiority of our approach compared to unstructured alternatives.