The ACRV Picking Benchmark (APB): A Robotic Shelf Picking Benchmark to Foster Reproducible Research
This provides a reproducible benchmark for researchers in robotics to test and compare complete picking systems, though it is incremental as it builds on existing challenge concepts.
The authors tackled the problem of limited reproducibility in robotic picking research by creating the ACRV Picking Benchmark (APB), a physical benchmark with 42 objects and standardized protocols that enables comparison of complete robotic systems, and they reported results from an open baseline system using a Baxter robot.
Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark (APB). Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of \emph{complete} robotic systems -- including perception and manipulation -- instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.