From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
This addresses the problem of robust zero-shot performance in robotic manipulation for researchers and practitioners, representing a strong specific gain rather than a foundational breakthrough.
The paper tackles the challenge of generalization in robotic manipulation for unseen scenarios by proposing FSD, a vision-language model that uses spatial relationship reasoning to guide actions, achieving a 40.6% success rate in simulation and 72% in real-world tasks, outperforming baselines by 30%.
Achieving generalization in robotic manipulation remains a critical challenge, particularly for unseen scenarios and novel tasks. Current Vision-Language-Action (VLA) models, while building on top of general Vision-Language Models (VLMs), still fall short of achieving robust zero-shot performance due to the scarcity and heterogeneity prevalent in embodied datasets. To address these limitations, we propose FSD (From Seeing to Doing), a novel vision-language model that generates intermediate representations through spatial relationship reasoning, providing fine-grained guidance for robotic manipulation. Our approach combines a hierarchical data pipeline for training with a self-consistency mechanism that aligns spatial coordinates with visual signals. Through extensive experiments, we comprehensively validated FSD's capabilities in both "seeing" and "doing," achieving outstanding performance across 8 benchmarks for general spatial reasoning and embodied reference abilities, as well as on our proposed more challenging benchmark VABench. We also verified zero-shot capabilities in robot manipulation, demonstrating significant performance improvements over baseline methods in both SimplerEnv and real robot settings. Experimental results show that FSD achieves 40.6% success rate in SimplerEnv and 72% success rate across 8 real-world tasks, outperforming the strongest baseline by 30%.