Active Surrogate Estimators: An Active Learning Approach to Label-Efficient Model Evaluation
This addresses the challenge of expensive labeling in model evaluation, particularly for deep learning, though it appears incremental as it builds on active testing methods.
The paper tackles the problem of label-efficient model evaluation by proposing Active Surrogate Estimators (ASEs), which use a surrogate-based approach and a novel acquisition strategy called XWED to actively learn errors, resulting in greater label-efficiency than the state-of-the-art for deep neural networks.
We propose Active Surrogate Estimators (ASEs), a new method for label-efficient model evaluation. Evaluating model performance is a challenging and important problem when labels are expensive. ASEs address this active testing problem using a surrogate-based estimation approach that interpolates the errors of points with unknown labels, rather than forming a Monte Carlo estimator. ASEs actively learn the underlying surrogate, and we propose a novel acquisition strategy, XWED, that tailors this learning to the final estimation task. We find that ASEs offer greater label-efficiency than the current state-of-the-art when applied to challenging model evaluation problems for deep neural networks.