MotionBenchMaker: A Tool to Generate and Benchmark Motion Planning Datasets
This tool addresses the problem of biased and time-consuming benchmarking for robotics researchers, though it is incremental as it builds on existing motion planning methods.
The paper tackles the challenge of evaluating motion planning algorithms for robotic manipulation by introducing MotionBenchMaker, an open-source tool that generates and benchmarks datasets, resulting in a suite of 40 prefabricated datasets for 5 robots in 8 environments to facilitate fair comparisons.
Recently, there has been a wealth of development in motion planning for robotic manipulation new motion planners are continuously proposed, each with their own unique strengths and weaknesses. However, evaluating new planners is challenging and researchers often create their own ad-hoc problems for benchmarking, which is time-consuming, prone to bias, and does not directly compare against other state-of-the-art planners. We present MotionBenchMaker, an open-source tool to generate benchmarking datasets for realistic robot manipulation problems. MotionBenchMaker is designed to be an extensible, easy-to-use tool that allows users to both generate datasets and benchmark them by comparing motion planning algorithms. Empirically, we show the benefit of using MotionBenchMaker as a tool to procedurally generate datasets which helps in the fair evaluation of planners. We also present a suite of 40 prefabricated datasets, with 5 different commonly used robots in 8 environments, to serve as a common ground to accelerate motion planning research.