Setup-Invariant Augmented Reality for Teaching by Demonstration with Surgical Robots
This work addresses the need for unsupervised training tools in robotic surgery education, enabling novices to practice outside the operating room with expert-informed guidance, though it is incremental as it builds on existing AR methods for a specific domain.
The researchers tackled the problem of AR systems for surgical robot training not accounting for differences in mentor and mentee robot configurations, and they developed dV-STEAR, which improved novice performance by increasing completion speed, reducing collision time, and improving success rates in tasks like ring-over-wire and pick-and-place.
Augmented reality (AR) is an effective tool in robotic surgery education as it combines exploratory learning with three-dimensional guidance. However, existing AR systems require expert supervision and do not account for differences in the mentor and mentee robot configurations. To enable novices to train outside the operating room while receiving expert-informed guidance, we present dV-STEAR: an open-source system that plays back task-aligned expert demonstrations without assuming identical setup joint positions between expert and novice. Pose estimation was rigorously quantified, showing a registration error of 3.86 (SD=2.01)mm. In a user study (N=24), dV-STEAR significantly improved novice performance on tasks from the Fundamentals of Laparoscopic Surgery. In a single-handed ring-over-wire task, dV-STEAR increased completion speed (p=0.03) and reduced collision time (p=0.01) compared to dry-lab training alone. During a pick-and-place task, it improved success rates (p=0.004). Across both tasks, participants using dV-STEAR exhibited significantly more balanced hand use and reported lower frustration levels. This work presents a novel educational tool implemented on the da Vinci Research Kit, demonstrates its effectiveness in teaching novices, and builds the foundation for further AR integration into robot-assisted surgery.