Understanding Uncertainty-based Active Learning Under Model Mismatch
This work addresses a critical issue for practitioners using active learning, revealing that UAL may fail in low-capacity models, which is incremental as it builds on existing UAL methods by highlighting a specific limitation.
The study tackled the problem of uncertainty-based active learning (UAL) under model mismatch, showing that UAL can perform worse than random sampling when the model has low capacity and cannot cover the ground truth, with empirical evidence supporting this finding.
Instead of randomly acquiring training data points, Uncertainty-based Active Learning (UAL) operates by querying the label(s) of pivotal samples from an unlabeled pool selected based on the prediction uncertainty, thereby aiming at minimizing the labeling cost for model training. The efficacy of UAL critically depends on the model capacity as well as the adopted uncertainty-based acquisition function. Within the context of this study, our analytical focus is directed toward comprehending how the capacity of the machine learning model may affect UAL efficacy. Through theoretical analysis, comprehensive simulations, and empirical studies, we conclusively demonstrate that UAL can lead to worse performance in comparison with random sampling when the machine learning model class has low capacity and is unable to cover the underlying ground truth. In such situations, adopting acquisition functions that directly target estimating the prediction performance may be beneficial for improving the performance of UAL.