Toloka Visual Question Answering Benchmark
This provides a new benchmark for evaluating visual question answering systems, though it is incremental as it focuses on dataset creation without novel methodological advances.
The authors introduced the Toloka Visual Question Answering dataset, containing 45,199 image-question pairs with bounding boxes, to benchmark machine learning systems against human performance in grounding visual question answering, but found that no model outperformed the non-expert crowdsourcing baseline in intersection over union scores.
In this paper, we present Toloka Visual Question Answering, a new crowdsourced dataset allowing comparing performance of machine learning systems against human level of expertise in the grounding visual question answering task. In this task, given an image and a textual question, one has to draw the bounding box around the object correctly responding to that question. Every image-question pair contains the response, with only one correct response per image. Our dataset contains 45,199 pairs of images and questions in English, provided with ground truth bounding boxes, split into train and two test subsets. Besides describing the dataset and releasing it under a CC BY license, we conducted a series of experiments on open source zero-shot baseline models and organized a multi-phase competition at WSDM Cup that attracted 48 participants worldwide. However, by the time of paper submission, no machine learning model outperformed the non-expert crowdsourcing baseline according to the intersection over union evaluation score.