LAMS: LLM-Driven Automatic Mode Switching for Assistive Teleoperation
This addresses inefficiencies in assistive teleoperation for users, though it appears incremental as it builds on existing mode-switching methods by leveraging LLMs.
The paper tackles the problem of cumbersome manual mode switching in teleoperating high-DoF robots with low-DoF controllers by introducing LAMS, an LLM-driven approach that automatically switches control modes based on task context, resulting in reduced manual switches and improved performance over time as validated in a user study with 10 participants.
Teleoperating high degrees-of-freedom (DoF) robotic manipulators via low-DoF controllers like joysticks often requires frequent switching between control modes, where each mode maps controller movements to specific robot actions. Manually performing this frequent switching can make teleoperation cumbersome and inefficient. On the other hand, existing automatic mode-switching solutions, such as heuristic-based or learning-based methods, are often task-specific and lack generalizability. In this paper, we introduce LLM-Driven Automatic Mode Switching (LAMS), a novel approach that leverages Large Language Models (LLMs) to automatically switch control modes based on task context. Unlike existing methods, LAMS requires no prior task demonstrations and incrementally improves by integrating user-generated mode-switching examples. We validate LAMS through an ablation study and a user study with 10 participants on complex, long-horizon tasks, demonstrating that LAMS effectively reduces manual mode switches, is preferred over alternative methods, and improves performance over time. The project website with supplementary materials is at https://lams-assistance.github.io/.