GraspLook: a VR-based Telemanipulation System with R-CNN-driven Augmentation of Virtual Environment
This addresses the need for more stable and convenient visual feedback in telemanipulation for medical applications, particularly in remote laboratory testing with hazardous reagents, though it is incremental as it builds on existing VR and R-CNN technologies.
The paper tackled the problem of teleoperating robots in medical settings by developing a VR-based system that uses R-CNN to detect instruments and display digital twins in a virtual environment, resulting in a 11% reduction in mental demand and 16% less effort compared to camera-based methods.
The teleoperation of robotic systems in medical applications requires stable and convenient visual feedback for the operator. The most accessible approach to delivering visual information from the remote area is using cameras to transmit a video stream from the environment. However, such systems are sensitive to the camera resolution, limited viewpoints, and cluttered environment bringing additional mental demands to the human operator. The paper proposes a novel system of teleoperation based on an augmented virtual environment (VE). The region-based convolutional neural network (R-CNN) is applied to detect the laboratory instrument and estimate its position in the remote environment to display further its digital twin in the VE, which is necessary for dexterous telemanipulation. The experimental results revealed that the developed system allows users to operate the robot smoother, which leads to a decrease in task execution time when manipulating test tubes. In addition, the participants evaluated the developed system as less mentally demanding (by 11%) and requiring less effort (by 16%) to accomplish the task than the camera-based teleoperation approach and highly assessed their performance in the augmented VE. The proposed technology can be potentially applied for conducting laboratory tests in remote areas when operating with infectious and poisonous reagents.