CURE-OR: Challenging Unreal and Real Environments for Object Recognition
This work addresses the problem of evaluating object recognition robustness under challenging conditions for researchers and developers, but it is incremental as it primarily provides a new dataset.
The authors introduced the CURE-OR dataset, containing 1,000,000 images of 100 objects under controlled challenging conditions, and used it to show that recognition APIs like Amazon Rekognition and Microsoft Azure Computer Vision significantly degrade in performance under these conditions, while also investigating the relationship between image quality and recognition.
In this paper, we introduce a large-scale, controlled, and multi-platform object recognition dataset denoted as Challenging Unreal and Real Environments for Object Recognition (CURE-OR). In this dataset, there are 1,000,000 images of 100 objects with varying size, color, and texture that are positioned in five different orientations and captured using five devices including a webcam, a DSLR, and three smartphone cameras in real-world (real) and studio (unreal) environments. The controlled challenging conditions include underexposure, overexposure, blur, contrast, dirty lens, image noise, resizing, and loss of color information. We utilize CURE-OR dataset to test recognition APIs-Amazon Rekognition and Microsoft Azure Computer Vision- and show that their performance significantly degrades under challenging conditions. Moreover, we investigate the relationship between object recognition and image quality and show that objective quality algorithms can estimate recognition performance under certain photometric challenging conditions. The dataset is publicly available at https://ghassanalregib.com/cure-or/.