RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
This addresses the challenge of spatial reasoning for robots using vision-language models, though it is incremental as it focuses on dataset creation rather than a new method.
The paper tackled the problem of spatial understanding in robotics by introducing RoboSpatial, a large-scale dataset with 1M images, 5k 3D scans, and 3M annotated spatial relationships, which improved model performance on tasks like spatial affordance prediction and robot manipulation.
Spatial understanding is a crucial capability that enables robots to perceive their surroundings, reason about their environment, and interact with it meaningfully. In modern robotics, these capabilities are increasingly provided by vision-language models. However, these models face significant challenges in spatial reasoning tasks, as their training data are based on general-purpose image datasets that often lack sophisticated spatial understanding. For example, datasets frequently do not capture reference frame comprehension, yet effective spatial reasoning requires understanding whether to reason from ego-, world-, or object-centric perspectives. To address this issue, we introduce RoboSpatial, a large-scale dataset for spatial understanding in robotics. It consists of real indoor and tabletop scenes, captured as 3D scans and egocentric images, and annotated with rich spatial information relevant to robotics. The dataset includes 1M images, 5k 3D scans, and 3M annotated spatial relationships, and the pairing of 2D egocentric images with 3D scans makes it both 2D- and 3D- ready. Our experiments show that models trained with RoboSpatial outperform baselines on downstream tasks such as spatial affordance prediction, spatial relationship prediction, and robot manipulation.