Towards Understanding the Behaviors of Optimal Deep Active Learning Algorithms
This research aims to provide a foundational understanding of optimal active learning strategies for researchers developing new AL algorithms, helping them identify shortcomings in current models.
This paper explores the characteristics of an optimal active learning (AL) algorithm by using simulated annealing to search for such an oracle across several tasks. The study provides qualitative and quantitative insights into the oracle's behavior, comparing it with existing heuristics and demonstrating consistent improvements to these heuristics based on one specific insight.
Active learning (AL) algorithms may achieve better performance with fewer data because the model guides the data selection process. While many algorithms have been proposed, there is little study on what the optimal AL algorithm looks like, which would help researchers understand where their models fall short and iterate on the design. In this paper, we present a simulated annealing algorithm to search for this optimal oracle and analyze it for several tasks. We present qualitative and quantitative insights into the behaviors of this oracle, comparing and contrasting them with those of various heuristics. Moreover, we are able to consistently improve the heuristics using one particular insight. We hope that our findings can better inform future active learning research. The code is available at https://github.com/YilunZhou/optimal-active-learning.