OffsetNet: Deep Learning for Localization in the Lung using Rendered Images
This addresses the need for accurate, real-time tool localization in lung surgery, which is incremental as it builds on existing methods to meet specific thresholds.
The paper tackles the problem of real-time bronchoscope localization in the lung for surgical navigation by introducing OffsetNet, a deep learning architecture that achieves an average position error of 1.4 mm in conserved regions and 2.4 mm in less conserved regions after training on recorded and simulated images.
Navigating surgical tools in the dynamic and tortuous anatomy of the lung's airways requires accurate, real-time localization of the tools with respect to the preoperative scan of the anatomy. Such localization can inform human operators or enable closed-loop control by autonomous agents, which would require accuracy not yet reported in the literature. In this paper, we introduce a deep learning architecture, called OffsetNet, to accurately localize a bronchoscope in the lung in real-time. After training on only 30 minutes of recorded camera images in conserved regions of a lung phantom, OffsetNet tracks the bronchoscope's motion on a held-out recording through these same regions at an update rate of 47 Hz and an average position error of 1.4 mm. Because this model performs poorly in less conserved regions, we augment the training dataset with simulated images from these regions. To bridge the gap between camera and simulated domains, we implement domain randomization and a generative adversarial network (GAN). After training on simulated images, OffsetNet tracks the bronchoscope's motion in less conserved regions at an average position error of 2.4 mm, which meets conservative thresholds required for successful tracking.