Cleaning the Pool: Progressive Filtering of Unlabeled Pools in Deep Active Learning
This addresses the challenge of selecting effective active learning strategies for practitioners, though it is incremental as it builds on existing ensemble methods.
The paper tackles the problem of suboptimal performance in active learning due to varying effectiveness of strategies across datasets and cycles by introducing REFINE, an ensemble method that combines multiple strategies through progressive filtering and coverage-based selection, achieving consistent outperformance over individual strategies and existing ensemble methods across 6 datasets and 3 models.
Existing active learning (AL) strategies capture fundamentally different notions of data value, e.g., uncertainty or representativeness. Consequently, the effectiveness of strategies can vary substantially across datasets, models, and even AL cycles. Committing to a single strategy risks suboptimal performance, as no single strategy dominates throughout the entire AL process. We introduce REFINE, an ensemble AL method that combines multiple strategies without knowing in advance which will perform best. In each AL cycle, REFINE operates in two stages: (1) Progressive filtering iteratively refines the unlabeled pool by considering an ensemble of AL strategies, retaining promising candidates capturing different notions of value. (2) Coverage-based selection then chooses a final batch from this refined pool, ensuring all previously identified notions of value are accounted for. Extensive experiments across 6 classification datasets and 3 foundation models show that REFINE consistently outperforms individual strategies and existing ensemble methods. Notably, progressive filtering serves as a powerful preprocessing step that improves the performance of any individual AL strategy applied to the refined pool, which we demonstrate on an audio spectrogram classification use case. Finally, the ensemble of REFINE can be easily extended with upcoming state-of-the-art AL strategies.