Real-Time Multimodal Activity-Aware Error Detection in Robot-Assisted Surgery
For the surgical robotics community, this work addresses the under-utilization of multimodal data and activity-aware descriptions to improve error detection, though it is an incremental advancement over existing methods.
The paper proposes a unified multimodal framework for executional error detection in robot-assisted surgery that integrates video, kinematics, and descriptive textual prompts, achieving up to 5% and 16.6% F1 score improvements over state-of-the-art baselines on JIGSAWS and SAR-RARP50 datasets.
Robot-assisted minimally invasive surgery improves surgical precision but introduces complexity, making technical error detection essential for ensuring patient safety. Current executional error detection methods using video data often overlook fine-grained contextual descriptions of activities and error types within the hierarchical structure of surgical procedures. They also under-utilize complementary multimodal information. We propose a unified framework for executional error detection that leverages multimodal input, including video, kinematics, and descriptive textual prompts. Through activity prompting, we integrate descriptive language in gesture-level activities, instrument-object interactions, and error definitions. We also introduce activity-aware visual embeddings derived from vision encoders pretrained on surgical activity labels to compare the effectiveness of contrastive language-image embeddings with traditional image-based embeddings for error detection. By seamlessly integrating kinematic data with video and textual modalities, our framework significantly improves error detection performance. Achieving up to 5\% and 16.6\% F1 score improvements over state-of-the-art baselines on the JIGSAWS and SAR-RARP50 datasets, respectively, we demonstrate the value of combining curated textual prompts with multimodal data for accurate error detection.