Benchmarking Simulated Robotic Manipulation through a Real World Dataset
This provides a standardized tool for researchers in robotics to assess simulation environments, though it is incremental as it builds on existing benchmarking practices.
The authors tackled the problem of evaluating simulated robotic manipulation by creating a benchmark that uses a real-world dataset to test simulation environments, applying it to PyBullet and V-Rep and publishing the results.
We present a benchmark to facilitate simulated manipulation; an attempt to overcome the obstacles of physical benchmarks through the distribution of a real world, ground truth dataset. Users are given various simulated manipulation tasks with assigned protocols having the objective of replicating the real world results of a recorded dataset. The benchmark comprises of a range of metrics used to characterise the successes of submitted environments whilst providing insight into their deficiencies. We apply our benchmark to two simulation environments, PyBullet and V-Rep, and publish the results. All materials required to benchmark an environment, including protocols and the dataset, can be found at the benchmarks' website https://research.csiro.au/robotics/manipulation-benchmark/.