A Robotic 3D Perception System for Operating Room Environment Awareness
This work addresses the need for context awareness in surgical robotics to support applications like workflow analysis and automation, representing an incremental advancement with a novel system for a specific domain.
The authors tackled the problem of enabling operating room scene understanding for robotic surgery by developing a 3D multi-view perception system using Time-of-Flight cameras on a da Vinci surgical robot, resulting in acceptable registration error (3.3% ± 1.4% of object-camera distance) and improved segmentation performance for less frequent classes (mIOU ≥ 0.013) compared to single-view methods.
Purpose: We describe a 3D multi-view perception system for the da Vinci surgical system to enable Operating room (OR) scene understanding and context awareness. Methods: Our proposed system is comprised of four Time-of-Flight (ToF) cameras rigidly attached to strategic locations on the daVinci Xi patient side cart (PSC). The cameras are registered to the robot's kinematic chain by performing a one-time calibration routine and therefore, information from all cameras can be fused and represented in one common coordinate frame. Based on this architecture, a multi-view 3D scene semantic segmentation algorithm is created to enable recognition of common and salient objects/equipment and surgical activities in a da Vinci OR. Our proposed 3D semantic segmentation method has been trained and validated on a novel densely annotated dataset that has been captured from clinical scenarios. Results: The results show that our proposed architecture has acceptable registration error ($3.3\%\pm1.4\%$ of object-camera distance) and can robustly improve scene segmentation performance (mean Intersection Over Union - mIOU) for less frequently appearing classes ($\ge 0.013$) compared to a single-view method. Conclusion: We present the first dynamic multi-view perception system with a novel segmentation architecture, which can be used as a building block technology for applications such as surgical workflow analysis, automation of surgical sub-tasks and advanced guidance systems.