Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments
This work addresses the need for better benchmarking of VLA models in robotics and AI, though it is incremental as it builds on existing benchmarks like Procgen.
The paper tackles the problem of systematically evaluating vision-language-action (VLA) models for zero-shot generalization in procedurally generated, out-of-distribution environments, finding that all models have significant limitations, with VLAs outperforming others and performance heavily influenced by factors like action representation and task complexity.
Vision-language-action (VLA) models represent an important step toward general-purpose robotic systems by integrating visual perception, language understanding, and action execution. However, systematic evaluation of these models, particularly their zero-shot generalization capabilities in procedurally out-of-distribution (OOD) environments, remains limited. In this paper, we introduce MultiNet v0.2, a comprehensive benchmark designed to evaluate and analyze the generalization performance of state-of-the-art VLMs and VLAs - including GPT-4o, GPT-4.1, OpenVLA, Pi0 Base, and Pi0 FAST - on diverse procedural tasks from the Procgen benchmark. Our analysis reveals several critical insights: (1) all evaluated models exhibit significant limitations in zero-shot generalization to OOD tasks, with performance heavily influenced by factors such as action representation and task complexity; (2) VLAs generally outperforms other models due to their robust architectural design; and (3) VLM variants demonstrate substantial improvements when constrained appropriately, highlighting the sensitivity of model performance to precise prompt engineering. We release our benchmark, evaluation framework, and findings to enable the assessment of future VLA models and identify critical areas for improvement in their application to out-of-distribution digital tasks.