SurgIRL: Towards Life-Long Learning for Surgical Automation by Incremental Reinforcement Learning
This work addresses the challenge of life-long learning for surgical automation, enabling robots to adapt to multiple tasks more efficiently, though it appears incremental in building on existing reinforcement learning methods.
The paper tackles the problem of surgical automation policies being limited in reusability across tasks by proposing SurgIRL, an incremental reinforcement learning framework that accumulates and reuses skills to solve multiple unseen tasks, with simulation experiments showing efficient automation of ten surgical tasks and successful sim-to-real transfers on the da Vinci Research Kit.
Surgical automation holds immense potential to improve the outcome and accessibility of surgery. Recent studies use reinforcement learning to learn policies that automate different surgical tasks. However, these policies are developed independently and are limited in their reusability when the task changes, making it more time-consuming when robots learn to solve multiple tasks. Inspired by how human surgeons build their expertise, we train surgical automation policies through Surgical Incremental Reinforcement Learning (SurgIRL). SurgIRL aims to (1) acquire new skills by referring to external policies (knowledge) and (2) accumulate and reuse these skills to solve multiple unseen tasks incrementally (incremental learning). Our SurgIRL framework includes three major components. We first define an expandable knowledge set containing heterogeneous policies that can be helpful for surgical tasks. Then, we propose Knowledge Inclusive Attention Network with mAximum Coverage Exploration (KIAN-ACE), which improves learning efficiency by maximizing the coverage of the knowledge set during the exploration process. Finally, we develop incremental learning pipelines based on KIAN-ACE to accumulate and reuse learned knowledge and solve multiple surgical tasks sequentially. Our simulation experiments show that KIAN-ACE efficiently learns to automate ten surgical tasks separately or incrementally. We also evaluate our learned policies on the da Vinci Research Kit (dVRK) and demonstrate successful sim-to-real transfers.