LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction
This work addresses the challenge of humanoid whole-body control for robotics, enabling more dynamic and complex tasks beyond quasi-static operations, though it appears incremental as it builds on existing vision-language-action models with a novel hierarchical approach.
The paper tackled the problem of enabling agile, whole-body behaviors in humanoid robots using vision-language-action models by introducing a new benchmark and a hierarchical framework called LeVERB, which achieved an 80% success rate on simple visual navigation tasks and 58.5% overall success rate, outperforming a baseline by 7.8 times.
Vision-language-action (VLA) models have demonstrated strong semantic understanding and zero-shot generalization, yet most existing systems assume an accurate low-level controller with hand-crafted action "vocabulary" such as end-effector pose or root velocity. This assumption confines prior work to quasi-static tasks and precludes the agile, whole-body behaviors required by humanoid whole-body control (WBC) tasks. To capture this gap in the literature, we start by introducing the first sim-to-real-ready, vision-language, closed-loop benchmark for humanoid WBC, comprising over 150 tasks from 10 categories. We then propose LeVERB: Latent Vision-Language-Encoded Robot Behavior, a hierarchical latent instruction-following framework for humanoid vision-language WBC, the first of its kind. At the top level, a vision-language policy learns a latent action vocabulary from synthetically rendered kinematic demonstrations; at the low level, a reinforcement-learned WBC policy consumes these latent verbs to generate dynamics-level commands. In our benchmark, LeVERB can zero-shot attain a 80% success rate on simple visual navigation tasks, and 58.5% success rate overall, outperforming naive hierarchical whole-body VLA implementation by 7.8 times.