A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation
This work addresses the problem of reducing annotation costs and improving evaluation precision for practitioners in machine learning and computer vision, though it is incremental as it builds on existing statistical methods.
The paper tackles the problem of expensive and imprecise model evaluation by proposing a statistical framework that uses stratification, sampling, and estimation to improve accuracy estimates. Experiments on computer vision datasets show that this method provides more precise estimates than simple random sampling, with efficiency gains of up to 10x.
Model performance evaluation is a critical and expensive task in machine learning and computer vision. Without clear guidelines, practitioners often estimate model accuracy using a one-time completely random selection of the data. However, by employing tailored sampling and estimation strategies, one can obtain more precise estimates and reduce annotation costs. In this paper, we propose a statistical framework for model evaluation that includes stratification, sampling, and estimation components. We examine the statistical properties of each component and evaluate their efficiency (precision). One key result of our work is that stratification via k-means clustering based on accurate predictions of model performance yields efficient estimators. Our experiments on computer vision datasets show that this method consistently provides more precise accuracy estimates than the traditional simple random sampling, even with substantial efficiency gains of 10x. We also find that model-assisted estimators, which leverage predictions of model accuracy on the unlabeled portion of the dataset, are generally more efficient than the traditional estimates based solely on the labeled data.