PushWorld: A benchmark for manipulation planning with tools and movable obstacles
This work addresses a gap in AI research for physical manipulation tasks, providing a benchmark to promote progress, though it is incremental as it builds on existing planning methods.
The authors tackled the challenge of physical reasoning tasks involving manipulation planning with movable obstacles and tools by introducing PushWorld, a benchmark with over 200 puzzles, and found that state-of-the-art algorithms, including a new heuristic that is 40 times faster, still perform below human-level.
While recent advances in artificial intelligence have achieved human-level performance in environments like Starcraft and Go, many physical reasoning tasks remain challenging for modern algorithms. To date, few algorithms have been evaluated on physical tasks that involve manipulating objects when movable obstacles are present and when tools must be used to perform the manipulation. To promote research on such tasks, we introduce PushWorld, an environment with simplistic physics that requires manipulation planning with both movable obstacles and tools. We provide a benchmark of more than 200 PushWorld puzzles in PDDL and in an OpenAI Gym environment. We evaluate state-of-the-art classical planning and reinforcement learning algorithms on this benchmark, and we find that these baseline results are below human-level performance. We then provide a new classical planning heuristic that solves the most puzzles among the baselines, and although it is 40 times faster than the best baseline planner, it remains below human-level performance.