Rebuilding Trust in Active Learning with Actionable Metrics
This work aims to increase the adoption of Active Learning in industry by providing metrics that offer interpretability and performance guarantees for practitioners.
This paper addresses the lack of industrial adoption of Active Learning (AL) by introducing actionable metrics to provide interpretability and guarantees for AL strategies. Through experiments on datasets like CIFAR100, Fashion-MNIST, and 20Newsgroups, the authors demonstrate that these metrics help practitioners understand and leverage AL performance.
Active Learning (AL) is an active domain of research, but is seldom used in the industry despite the pressing needs. This is in part due to a misalignment of objectives, while research strives at getting the best results on selected datasets, the industry wants guarantees that Active Learning will perform consistently and at least better than random labeling. The very one-off nature of Active Learning makes it crucial to understand how strategy selection can be carried out and what drives poor performance (lack of exploration, selection of samples that are too hard to classify, ...). To help rebuild trust of industrial practitioners in Active Learning, we present various actionable metrics. Through extensive experiments on reference datasets such as CIFAR100, Fashion-MNIST, and 20Newsgroups, we show that those metrics brings interpretability to AL strategies that can be leveraged by the practitioner.