Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models
This work addresses the data scarcity issue for failure detection in robotic manipulation, enabling more reliable systems, though it is incremental as it builds on existing Vision-Language Models.
The paper tackled the problem of limited failure data for robust robotic manipulation by proposing an automatic robot failure synthesis approach to generate diverse planning and execution failures, and introduced Guardian, a Vision-Language Model that achieved state-of-the-art performance on new benchmarks and improved task success rates in simulation and real robots.
Robust robotic manipulation requires reliable failure detection and recovery. Although current Vision-Language Models (VLMs) show promise, their accuracy and generalization are limited by the scarcity of failure data. To address this data gap, we propose an automatic robot failure synthesis approach that procedurally perturbs successful trajectories to generate diverse planning and execution failures. This method produces not only binary classification labels but also fine-grained failure categories and step-by-step reasoning traces in both simulation and the real world. With it, we construct three new failure detection benchmarks: RLBench-Fail, BridgeDataV2-Fail, and UR5-Fail, substantially expanding the diversity and scale of existing failure datasets. We then train Guardian, a VLM with multi-view images for detailed failure reasoning and detection. Guardian achieves state-of-the-art performance on both existing and newly introduced benchmarks. It also effectively improves task success rates when integrated into a state-of-the-art manipulation system in simulation and real robots, demonstrating the impact of our generated failure data. Code, Data, and Models available at https://www.di.ens.fr/willow/research/guardian/.