CapsDT: Diffusion-Transformer for Capsule Robot Manipulation
This work addresses the challenge of intuitive and efficient human-robot interaction for medical diagnostics and treatments in endoscopy robotics, representing an incremental advancement by adapting existing methods to a new domain.
The paper tackled the problem of applying Vision-Language-Action models to endoscopy capsule robot manipulation in the stomach, achieving a 26.25% success rate in real-world simulation and state-of-the-art performance on various endoscopy tasks.
Vision-Language-Action (VLA) models have emerged as a prominent research area, showcasing significant potential across a variety of applications. However, their performance in endoscopy robotics, particularly endoscopy capsule robots that perform actions within the digestive system, remains unexplored. The integration of VLA models into endoscopy robots allows more intuitive and efficient interactions between human operators and medical devices, improving both diagnostic accuracy and treatment outcomes. In this work, we design CapsDT, a Diffusion Transformer model for capsule robot manipulation in the stomach. By processing interleaved visual inputs, and textual instructions, CapsDT can infer corresponding robotic control signals to facilitate endoscopy tasks. In addition, we developed a capsule endoscopy robot system, a capsule robot controlled by a robotic arm-held magnet, addressing different levels of four endoscopy tasks and creating corresponding capsule robot datasets within the stomach simulator. Comprehensive evaluations on various robotic tasks indicate that CapsDT can serve as a robust vision-language generalist, achieving state-of-the-art performance in various levels of endoscopy tasks while achieving a 26.25% success rate in real-world simulation manipulation.