ObjectNet Dataset: Reanalysis and Correction
This work corrects a methodological flaw in a prior study on generalization testing for object recognition models, but it is incremental as it builds on existing findings.
The paper reanalyzes the ObjectNet dataset, identifying a major issue with applying object recognizers to scenes containing multiple objects rather than isolated objects, which recovers 10-15% of the performance loss without test-time augmentation, though deep models still suffer drastically on this dataset.
Recently, Barbu et al introduced a dataset called ObjectNet which includes objects in daily life situations. They showed a dramatic performance drop of the state of the art object recognition models on this dataset. Due to the importance and implications of their results regarding generalization ability of deep models, we take a second look at their findings. We highlight a major problem with their work which is applying object recognizers to the scenes containing multiple objects rather than isolated objects. The latter results in around 20-30% performance gain using our code. Compared with the results reported in the ObjectNet paper, we observe that around 10-15 % of the performance loss can be recovered, without any test time data augmentation. In accordance with Barbu et al.'s conclusions, however, we also conclude that deep models suffer drastically on this dataset. Thus, we believe that ObjectNet remains a challenging dataset for testing the generalization power of models beyond datasets on which they have been trained.