Overcoming Overconfidence for Active Learning
This work addresses the issue of constrained labeling budgets in AI by improving active learning, though it appears incremental as it builds on existing methods to handle overconfidence.
The paper tackles the problem of model overconfidence in active learning, which arises from limited labeled data, and introduces two novel methods—Cross-Mix-and-Mix (CMaM) for calibration and Ranked Margin Sampling (RankedMS) for selection—that demonstrate efficient data selection by alleviating overconfidence in experiments.
It is not an exaggeration to say that the recent progress in artificial intelligence technology depends on large-scale and high-quality data. Simultaneously, a prevalent issue exists everywhere: the budget for data labeling is constrained. Active learning is a prominent approach for addressing this issue, where valuable data for labeling is selected through a model and utilized to iteratively adjust the model. However, due to the limited amount of data in each iteration, the model is vulnerable to bias; thus, it is more likely to yield overconfident predictions. In this paper, we present two novel methods to address the problem of overconfidence that arises in the active learning scenario. The first is an augmentation strategy named Cross-Mix-and-Mix (CMaM), which aims to calibrate the model by expanding the limited training distribution. The second is a selection strategy named Ranked Margin Sampling (RankedMS), which prevents choosing data that leads to overly confident predictions. Through various experiments and analyses, we are able to demonstrate that our proposals facilitate efficient data selection by alleviating overconfidence, even though they are readily applicable.