Spatial Traces: Enhancing VLA Models with Spatial-Temporal Understanding
This incremental improvement addresses spatial-temporal understanding in VLA models, benefiting applications in virtual and real-world environments where data collection is challenging.
The paper tackled the problem of enhancing Vision-Language-Action models by integrating spatial and temporal understanding through visual prompting, resulting in a 4% increase in tasks solved compared to SpatialVLA and 19% compared to TraceVLA in experiments.
Vision-Language-Action models have demonstrated remarkable capabilities in predicting agent movements within virtual environments and real-world scenarios based on visual observations and textual instructions. Although recent research has focused on enhancing spatial and temporal understanding independently, this paper presents a novel approach that integrates both aspects through visual prompting. We introduce a method that projects visual traces of key points from observations onto depth maps, enabling models to capture both spatial and temporal information simultaneously. The experiments in SimplerEnv show that the mean number of tasks successfully solved increased for 4% compared to SpatialVLA and 19% compared to TraceVLA. Furthermore, we show that this enhancement can be achieved with minimal training data, making it particularly valuable for real-world applications where data collection is challenging. The project page is available at https://ampiromax.github.io/ST-VLA.