A Cross-Domain Benchmark for Active Learning
This provides a standardized benchmark for active learning researchers to improve evaluation reliability, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of poor generalization and limited experimental repetitions in active learning research by introducing CDALBench, a cross-domain benchmark covering computer vision, natural language processing, and tabular learning, which showed that method superiority varies across domains and that conducting only three runs can lead to inconsistent performance results.
Active Learning (AL) deals with identifying the most informative samples for labeling to reduce data annotation costs for supervised learning tasks. AL research suffers from the fact that lifts from literature generalize poorly and that only a small number of repetitions of experiments are conducted. To overcome these obstacles, we propose CDALBench, the first active learning benchmark which includes tasks in computer vision, natural language processing and tabular learning. Furthermore, by providing an efficient, greedy oracle, CDALBench can be evaluated with 50 runs for each experiment. We show, that both the cross-domain character and a large amount of repetitions are crucial for sophisticated evaluation of AL research. Concretely, we show that the superiority of specific methods varies over the different domains, making it important to evaluate Active Learning with a cross-domain benchmark. Additionally, we show that having a large amount of runs is crucial. With only conducting three runs as often done in the literature, the superiority of specific methods can strongly vary with the specific runs. This effect is so strong, that, depending on the seed, even a well-established method's performance can be significantly better and significantly worse than random for the same dataset.