Learning to See Forces: Surgical Force Prediction with RGB-Point Cloud Temporal Convolutional Networks
This addresses the limitation of haptic feedback in robotic surgery, offering a low-cost alternative to hardware solutions for surgeons, though it is incremental as it builds on existing visual cue and deep learning approaches.
The paper tackles the problem of obtaining haptic feedback in robotic surgery by proposing a deep learning method to infer surgical forces from endoscopic video, achieving a mean absolute error of 0.814 N in an ex vivo liver study.
Robotic surgery has been proven to offer clear advantages during surgical procedures, however, one of the major limitations is obtaining haptic feedback. Since it is often challenging to devise a hardware solution with accurate force feedback, we propose the use of "visual cues" to infer forces from tissue deformation. Endoscopic video is a passive sensor that is freely available, in the sense that any minimally-invasive procedure already utilizes it. To this end, we employ deep learning to infer forces from video as an attractive low-cost and accurate alternative to typically complex and expensive hardware solutions. First, we demonstrate our approach in a phantom setting using the da Vinci Surgical System affixed with an OptoForce sensor. Second, we then validate our method on an ex vivo liver organ. Our method results in a mean absolute error of 0.814 N in the ex vivo study, suggesting that it may be a promising alternative to hardware based surgical force feedback in endoscopic procedures.