Target-Bench: Can World Models Achieve Mapless Path Planning with Semantic Targets?
This addresses the need for standardized evaluation in robotics and AI by providing a benchmark for world models, though it is incremental as it focuses on a specific domain rather than broad advancements.
The paper tackles the problem of evaluating world models for mapless path planning toward semantic targets in real-world environments by introducing Target-Bench, a benchmark with 450 robot-collected video sequences, and finds that fine-tuning a model improves performance by over 400% to a score of 0.345, surpassing the best off-the-shelf model at 0.299.
While recent world models generate highly realistic videos, their ability to perform robot path planning remains unclear and unquantified. We introduce Target-Bench, the first benchmark specifically designed to evaluate world models on mapless path planning toward semantic targets in real-world environments. Target-Bench provides 450 robot-collected video sequences spanning 45 semantic categories with SLAM-based ground truth trajectories. Our evaluation pipeline recovers camera motion from generated videos and measures planning performance using five complementary metrics that quantify target-reaching capability, trajectory accuracy, and directional consistency. We evaluate state-of-the-art models including Sora 2, Veo 3.1, and the Wan series. The best off-the-shelf model (Wan2.2-Flash) achieves only 0.299 overall score, revealing significant limitations in current world models for robotic planning tasks. We show that fine-tuning an open-source 5B-parameter model on only 325 scenarios from our dataset achieves 0.345 overall score -- an improvement of more than 400% over its base version (0.066) and 15% higher than the best off-the-shelf model. We will open-source the code and dataset.