Bayesian Estimate of Mean Proper Scores for Diversity-Enhanced Active Learning
This work addresses the need for more efficient and diverse sampling in active learning, particularly for applications in text and image classification, though it is incremental as it builds on existing Bayesian frameworks.
The paper tackled the problem of improving sampling efficiency in active learning by proposing Bayesian Estimate of Mean Proper Scores (BEMPS) to estimate increases in proper scores like log probability or negative mean square error, and experiments showed it consistently outperformed other methods with robust acquisition functions and well-calibrated classifiers.
The effectiveness of active learning largely depends on the sampling efficiency of the acquisition function. Expected Loss Reduction (ELR) focuses on a Bayesian estimate of the reduction in classification error, and more general costs fit in the same framework. We propose Bayesian Estimate of Mean Proper Scores (BEMPS) to estimate the increase in strictly proper scores such as log probability or negative mean square error within this framework. We also prove convergence results for this general class of costs. To facilitate better experimentation with the new acquisition functions, we develop a complementary batch AL algorithm that encourages diversity in the vector of expected changes in scores for unlabeled data. To allow high-performance classifiers, we combine deep ensembles, and dynamic validation set construction on pretrained models, and further speed up the ensemble process with the idea of Monte Carlo Dropout. Extensive experiments on both texts and images show that the use of mean square error and log probability with BEMPS yields robust acquisition functions and well-calibrated classifiers, and consistently outperforms the others tested. The advantages of BEMPS over the others are further supported by a set of qualitative analyses, where we visualise their sampling behaviour using data maps and t-SNE plots.