Diagnose, Correct, and Learn from Manipulation Failures via Visual Symbols
This work addresses the problem of robotic manipulation failure diagnosis and correction for researchers and practitioners in robotics and AI, though it is incremental as it builds on existing VLA and VLM approaches.
The paper tackles the limitation of Vision-Language-Action models in diagnosing and learning from robotic manipulation failures by introducing ViFailback, a framework that uses visual symbols for efficient annotation and correction guidance, resulting in a dataset of 58,126 VQA pairs and 5,202 real-world trajectories, with their ViFailback-8B model showing significant performance improvements on a new benchmark and enabling real-world failure recovery.
Vision-Language-Action (VLA) models have recently achieved remarkable progress in robotic manipulation, yet they remain limited in failure diagnosis and learning from failures. Additionally, existing failure datasets are mostly generated programmatically in simulation, which limits their generalization to the real world. In light of these, we introduce ViFailback, a framework designed to diagnose robotic manipulation failures and provide both textual and visual correction guidance. Our framework utilizes explicit visual symbols to enhance annotation efficiency. We further release the ViFailback dataset, a large-scale collection of 58,126 Visual Question Answering (VQA) pairs along with their corresponding 5,202 real-world manipulation trajectories. Based on the dataset, we establish ViFailback-Bench, a benchmark of 11 fine-grained VQA tasks designed to assess the failure diagnosis and correction abilities of Vision-Language Models (VLMs), featuring ViFailback-Bench Lite for closed-ended and ViFailback-Bench Hard for open-ended evaluation. To demonstrate the effectiveness of our framework, we built the ViFailback-8B VLM, which not only achieves significant overall performance improvement on ViFailback-Bench but also generates visual symbols for corrective action guidance. Finally, by integrating ViFailback-8B with a VLA model, we conduct real-world robotic experiments demonstrating its ability to assist the VLA model in recovering from failures. Project Website: https://x1nyuzhou.github.io/vifailback.github.io/