Adapt as You Say: Online Interactive Bimanual Skill Adaptation via Human Language Feedback
This work addresses the problem of enabling general-purpose robots to adapt to evolving conditions in human environments, offering a novel method for non-expert customization, though it is incremental in building on existing adaptation techniques.
The paper tackles the challenge of adapting high-dimensional bimanual robot skills to novel task variations at deployment, presenting BiSAIL, a framework that enables zero-shot online adaptation via interactive language feedback, achieving significant improvements in adaptability, generalization, and scalability across real-robot experiments.
Developing general-purpose robots capable of autonomously operating in human living environments requires the ability to adapt to continuously evolving task conditions. However, adapting high-dimensional coordinated bimanual skills to novel task variations at deployment remains a fundamental challenge. In this work, we present BiSAIL (Bimanual Skill Adaptation via Interactive Language), a novel framework that enables zero-shot online adaptation of offline-learned bimanual skills through interactive language feedback. The key idea of BiSAIL is to adopt a hierarchical reason-then-modulate paradigm, which first infers generalized adaptation objectives from multimodal task variations, and then adapts bimanual motions via diffusion modulation to achieve the inferred objectives. Extensive real-robot experiments across six bimanual tasks and two dual-arm platforms demonstrate that BiSAIL significantly outperforms existing methods in human-in-the-loop adaptability, task generalization and cross-embodiment scalability. This work enables the development of adaptive bimanual assistants that can be flexibly customized by non-expert users via intuitive verbal corrections. Experimental videos and code are available at https://rip4kobe.github.io/BiSAIL/.