Fault-Tolerant Evaluation for Sample-Efficient Model Performance Estimators
This addresses the challenge of reliable validation for third-party AI models in dynamic applications, though it is incremental as it improves evaluation methods rather than introducing a new paradigm.
The paper tackles the problem of evaluating sample-efficient model performance estimators in low-variance settings, where existing methods like RMSE and p-value tests fail, by proposing a fault-tolerant framework with an adjustable tolerance level ε that integrates bias and variance, and experiments on real-world datasets show it provides comprehensive insights.
In the era of Model-as-a-Service, organizations increasingly rely on third-party AI models for rapid deployment. However, the dynamic nature of emerging AI applications, the continual introduction of new datasets, and the growing number of models claiming superior performance make efficient and reliable validation of model services increasingly challenging. This motivates the development of sample-efficient performance estimators, which aim to estimate model performance by strategically selecting instances for labeling, thereby reducing annotation cost. Yet existing evaluation approaches often fail in low-variance settings: RMSE conflates bias and variance, masking persistent bias when variance is small, while p-value based tests become hypersensitive, rejecting adequate estimators for negligible deviations. To address this, we propose a fault-tolerant evaluation framework that integrates bias and variance considerations within an adjustable tolerance level ${\varepsilon}$, enabling the evaluation of performance estimators within practically acceptable error margins. We theoretically show that proper calibration of ${\varepsilon}$ ensures reliable evaluation across different variance regimes, and we further propose an algorithm that automatically optimizes and selects ${\varepsilon}$. Experiments on real-world datasets demonstrate that our framework provides comprehensive and actionable insights into estimator behavior.