Chunchun Li

Touqeer Ahmad, Mohsen Jafarzadeh, Akshay Raj Dhamija et al.

There exists a distribution discrepancy between training and testing, in the way images are fed to modern CNNs. Recent work tried to bridge this gap either by fine-tuning or re-training the network at different resolutions. However re-training a network is rarely cheap and not always viable. To this end, we propose a simple solution to address the train-test distributional shift and enhance the performance of pre-trained models -- which commonly ship as a package with deep learning platforms \eg, PyTorch. Specifically, we demonstrate that running inference on the center crop of an image is not always the best as important discriminatory information may be cropped-off. Instead we propose to combine results for multiple random crops for a test image. This not only matches the train time augmentation but also provides the full coverage of the input image. We explore combining representation of random crops through averaging at different levels \ie, deep feature level, logit level, and softmax level. We demonstrate that, for various families of modern deep networks, such averaging results in better validation accuracy compared to using a single central crop per image. The softmax averaging results in the best performance for various pre-trained networks without requiring any re-training or fine-tuning whatsoever. On modern GPUs with batch processing, the paper's approach to inference of pre-trained networks, is essentially free as all images in a batch can all be processed at once.

Akshay Raj Dhamija, Touqeer Ahmad, Jonathan Schwan et al.

This paper identifies the flaws in existing open-world learning approaches and attempts to provide a complete picture in the form of \textbf{True Open-World Learning}. We accomplish this by proposing a comprehensive generalize-able open-world learning protocol capable of evaluating various components of open-world learning in an operational setting. We argue that in true open-world learning, the underlying feature representation should be learned in a self-supervised manner. Under this self-supervised feature representation, we introduce the problem of detecting unknowns as samples belonging to Out-of-Label space. We differentiate between Out-of-Label space detection and the conventional Out-of-Distribution detection depending upon whether the unknowns being detected belong to the native-world (same as feature representation) or a new-world, respectively. Our unifying open-world learning framework combines three individual research dimensions, which typically have been explored independently, i.e., Incremental Learning, Out-of-Distribution detection and Open-World Learning. Starting from a self-supervised feature space, an open-world learner has the ability to adapt and specialize its feature space to the classes in each incremental phase and hence perform better without incurring any significant overhead, as demonstrated by our experimental results. The incremental learning component of our pipeline provides the new state-of-the-art on established ImageNet-100 protocol. We also demonstrate the adaptability of our approach by showing how it can work as a plug-in with any of the self-supervised feature representation methods.

Mohsen Jafarzadeh, Akshay Raj Dhamija, Steve Cruz et al.

In open-world learning, an agent starts with a set of known classes, detects, and manages things that it does not know, and learns them over time from a non-stationary stream of data. Open-world learning is related to but also distinct from a multitude of other learning problems and this paper briefly analyzes the key differences between a wide range of problems including incremental learning, generalized novelty discovery, and generalized zero-shot learning. This paper formalizes various open-world learning problems including open-world learning without labels. These open-world problems can be addressed with modifications to known elements, we present a new framework that enables agents to combine various modules for novelty-detection, novelty-characterization, incremental learning, and instance management to learn new classes from a stream of unlabeled data in an unsupervised manner, survey how to adapt a few state-of-the-art techniques to fit the framework and use them to define seven baselines for performance on the open-world learning without labels problem. We then discuss open-world learning quality and analyze how that can improve instance management. We also discuss some of the general ambiguity issues that occur in open-world learning without labels.

Mohsen Jafarzadeh, Touqeer Ahmad, Akshay Raj Dhamija et al.

Image classification in the open-world must handle out-of-distribution (OOD) images. Systems should ideally reject OOD images, or they will map atop of known classes and reduce reliability. Using open-set classifiers that can reject OOD inputs can help. However, optimal accuracy of open-set classifiers depend on the frequency of OOD data. Thus, for either standard or open-set classifiers, it is important to be able to determine when the world changes and increasing OOD inputs will result in reduced system reliability. However, during operations, we cannot directly assess accuracy as there are no labels. Thus, the reliability assessment of these classifiers must be done by human operators, made more complex because networks are not 100% accurate, so some failures are to be expected. To automate this process, herein, we formalize the open-world recognition reliability problem and propose multiple automatic reliability assessment policies to address this new problem using only the distribution of reported scores/probability data. The distributional algorithms can be applied to both classic classifiers with SoftMax as well as the open-world Extreme Value Machine (EVM) to provide automated reliability assessment. We show that all of the new algorithms significantly outperform detection using the mean of SoftMax.