VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Models
This work addresses the problem of inconsistent and opaque improvements in VLA models for robotics researchers, providing a systematic analysis to guide future development.
The paper tackled the challenge of understanding performance gains in Vision-Language-Action (VLA) models by introducing VLA-OS, a unified architecture series, and conducting controlled experiments across diverse settings. The results showed that visually grounded planning representations generally outperform language-based ones, and the Hierarchical-VLA paradigm achieves superior or comparable performance in various metrics, though with slower speeds.
Recent studies on Vision-Language-Action (VLA) models have shifted from the end-to-end action-generation paradigm toward a pipeline involving task planning followed by action generation, demonstrating improved performance on various complex, long-horizon manipulation tasks. However, existing approaches vary significantly in terms of network architectures, planning paradigms, representations, and training data sources, making it challenging for researchers to identify the precise sources of performance gains and components to be further improved. To systematically investigate the impacts of different planning paradigms and representations isolating from network architectures and training data, in this paper, we introduce VLA-OS, a unified VLA architecture series capable of various task planning paradigms, and design a comprehensive suite of controlled experiments across diverse object categories (rigid and deformable), visual modalities (2D and 3D), environments (simulation and real-world), and end-effectors (grippers and dexterous hands). Our results demonstrate that: 1) visually grounded planning representations are generally better than language planning representations; 2) the Hierarchical-VLA paradigm generally achieves superior or comparable performance than other paradigms on task performance, pretraining, generalization ability, scalability, and continual learning ability, albeit at the cost of slower training and inference speeds.