4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration
This work addresses pretraining efficiency for robotic manipulation tasks, representing an incremental improvement with novel components for a specific bottleneck.
The paper tackles the challenge of leveraging diverse robotic data for pretraining by addressing coordinate system and state chaos that disperses conditional action distributions. Their 4D-VLA approach integrates depth and temporal information with memory bank sampling, achieving a significant increase in success rate over OpenVLA in simulated and real-world experiments.
Leveraging diverse robotic data for pretraining remains a critical challenge. Existing methods typically model the dataset's action distribution using simple observations as inputs. However, these inputs are often incomplete, resulting in a dispersed conditional action distribution-an issue we refer to as coordinate system chaos and state chaos. This inconsistency significantly hampers pretraining efficiency. To address this, we propose 4D-VLA, a novel approach that effectively integrates 4D information into the input to mitigate these sources of chaos. Our model introduces depth and temporal information into visual features with sequential RGB-D inputs, aligning the coordinate systems of the robot and the scene. This alignment endows the model with strong spatiotemporal reasoning capabilities while minimizing training overhead. Additionally, we introduce memory bank sampling, a frame sampling strategy designed to extract informative frames from historical images, further improving effectiveness and efficiency. Experimental results demonstrate that our pretraining method and architectural components substantially enhance model performance. In both simulated and real-world experiments, our model achieves a significant increase in success rate over OpenVLA. To further assess spatial perception and generalization to novel views, we introduce MV-Bench, a multi-view simulation benchmark. Our model consistently outperforms existing methods, demonstrating stronger spatial understanding and adaptability.