HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers
This work addresses efficiency and performance limitations in robotic control for dynamic tasks, representing an incremental improvement over existing methods.
The paper tackles the high computational cost and latency of large Vision-Language-Action models in robotic control by proposing HiRT, a hierarchical framework that enables flexible frequency trade-offs, resulting in doubled control frequency in static tasks and improved success rates from 48% to 75% in dynamic manipulation tasks.
Large Vision-Language-Action (VLA) models, leveraging powerful pre trained Vision-Language Models (VLMs) backends, have shown promise in robotic control due to their impressive generalization ability. However, the success comes at a cost. Their reliance on VLM backends with billions of parameters leads to high computational costs and inference latency, limiting the testing scenarios to mainly quasi-static tasks and hindering performance in dynamic tasks requiring rapid interactions. To address these limitations, this paper proposes HiRT, a Hierarchical Robot Transformer framework that enables flexible frequency and performance trade-off. HiRT keeps VLMs running at low frequencies to capture temporarily invariant features while enabling real-time interaction through a high-frequency vision-based policy guided by the slowly updated features. Experiment results in both simulation and real-world settings demonstrate significant improvements over baseline methods. Empirically, in static tasks, we double the control frequency and achieve comparable success rates. Additionally, on novel real-world dynamic ma nipulation tasks which are challenging for previous VLA models, HiRT improves the success rate from 48% to 75%.