TRACE: Textual Reasoning for Affordance Coordinate Extraction
This work addresses the challenge of improving precision and interpretability in VLM-based robot control for robotics applications, representing a novel method for a known bottleneck.
The paper tackles the problem of Vision-Language Models (VLMs) struggling to translate high-level instructions into precise spatial affordances for robotic manipulation by introducing TRACE, a methodology that integrates a textual Chain of Reasoning into affordance prediction, resulting in state-of-the-art performance with 48.1% accuracy on the Where2Place benchmark (a 9.6% relative improvement) and 55.0% on a more challenging subset.
Vision-Language Models (VLMs) struggle to translate high-level instructions into the precise spatial affordances required for robotic manipulation. While visual Chain-of-Thought (CoT) methods exist, they are often computationally intensive. In this work, we introduce TRACE (Textual Reasoning for Affordance Coordinate Extraction), a novel methodology that integrates a textual Chain of Reasoning (CoR) into the affordance prediction process. We use this methodology to create the TRACE dataset, a large-scale collection created via an autonomous pipeline that pairs instructions with explicit textual rationales. By fine-tuning a VLM on this data, our model learns to externalize its spatial reasoning before acting. Our experiments show that our TRACE-tuned model achieves state-of-the-art performance, reaching 48.1% accuracy on the primary Where2Place (W2P) benchmark (a 9.6% relative improvement) and 55.0% on the more challenging W2P(h) subset. Crucially, an ablation study demonstrates that performance scales directly with the amount of reasoning data used, confirming the CoR's effectiveness. Furthermore, analysis of the model's attention maps reveals an interpretable reasoning process where focus shifts dynamically across reasoning steps. This work shows that training VLMs to generate a textual CoR is an effective and robust strategy for enhancing the precision, reliability, and interpretability of VLM-based robot control. Our dataset and code are available at https://github.com/jink-ucla/TRACE