Continual Improvement of Threshold-Based Novelty Detection
This addresses the issue of deploying continual learners in realistic environments where novel categories are encountered without explicit notification, though it appears incremental as it builds on existing threshold-based methods.
The paper tackles the problem of neural networks struggling to detect unseen classes in dynamic, open-world continual learning by proposing a method to automatically select thresholds for novelty detection, resulting in improved total accuracy on MNIST, Fashion MNIST, and CIFAR-10 datasets.
When evaluated in dynamic, open-world situations, neural networks struggle to detect unseen classes. This issue complicates the deployment of continual learners in realistic environments where agents are not explicitly informed when novel categories are encountered. A common family of techniques for detecting novelty relies on thresholds of similarity between observed data points and the data used for training. However, these methods often require manually specifying (ahead of time) the value of these thresholds, and are therefore incapable of adapting to the nature of the data. We propose a new method for automatically selecting these thresholds utilizing a linear search and leave-one-out cross-validation on the ID classes. We demonstrate that this novel method for selecting thresholds results in improved total accuracy on MNIST, Fashion MNIST, and CIFAR-10.