Oracle MCG: A first peek into COCO Detection Challenges
This provides an initial benchmark for researchers to gauge their methods against the challenging COCO detection dataset, though it is incremental as it uses existing proposals without refinement.
The paper addresses the lack of context for evaluating object detection techniques on the new COCO benchmark by simulating an oracle detector that selects the best proposals from a state-of-the-art method, achieving AP=0.292 for segmented objects and AP=0.317 for bounding boxes, indicating the dataset's high difficulty.
The recently presented COCO detection challenge will most probably be the reference benchmark in object detection in the next years. COCO is two orders of magnitude larger than Pascal and has four times the number of categories; so in all likelihood researchers will be faced with a number of new challenges. At this point, without any finished round of the competition, it is difficult for researchers to put their techniques in context, or in other words, to know how good their results are. In order to give a little context, this note evaluates a hypothetical object detector consisting in an oracle picking the best object proposal from a state-of-the-art technique. This oracle achieves a AP=0.292 in segmented objects and AP=0.317 in bounding boxes, showing that indeed the database is challenging, given that this value is the best one can expect if working on object proposals without refinement.