pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning
This addresses a critical limitation in AI for robotics and augmented reality by enabling zero-shot spatial understanding without fine-tuning, though it is incremental as it builds on existing MLLM and spatial tool capabilities.
The paper tackles the problem of multi-modal large language models struggling with 3D spatial reasoning by introducing pySpatial, a visual programming framework that generates Python code to interface with spatial tools, converting 2D images into explorable 3D scenes for explicit reasoning; it outperforms GPT-4.1-mini by 12.94% on the MindCube benchmark and enables successful real-world indoor navigation.
Multi-modal Large Language Models (MLLMs) have demonstrated strong capabilities in general-purpose perception and reasoning, but they still struggle with tasks that require spatial understanding of the 3D world. To address this, we introduce pySpatial, a visual programming framework that equips MLLMs with the ability to interface with spatial tools via Python code generation. Given an image sequence and a natural-language query, the model composes function calls to spatial tools including 3D reconstruction, camera-pose recovery, novel-view rendering, etc. These operations convert raw 2D inputs into an explorable 3D scene, enabling MLLMs to reason explicitly over structured spatial representations. Notably, pySpatial requires no gradient-based fine-tuning and operates in a fully zero-shot setting. Experimental evaluations on the challenging MindCube and Omni3D-Bench benchmarks demonstrate that our framework pySpatial consistently surpasses strong MLLM baselines; for instance, it outperforms GPT-4.1-mini by 12.94% on MindCube. Furthermore, we conduct real-world indoor navigation experiments where the robot can successfully traverse complex environments using route plans generated by pySpatial, highlighting the practical effectiveness of our approach.