Dynamically Allocating Evaluation Effort for Model Ranking
For NLP researchers and organizers of model competitions, this work reduces the cost of human evaluation while improving the ability to identify top models.
The paper formalizes multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup, proposing an adaptive sampling algorithm that focuses annotation budget on competitive models. They prove optimality and show improved discrimination between top-performing models, making evaluations faster and cheaper.
While human evaluation is the gold standard in many NLP tasks, it suffers from prohibitive costs and poor scalability. When identifying top-performing models, typical evaluation protocols waste effort by exhaustively evaluating all models on the entire benchmark, a safe but inefficient approach. In this work, we formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms, where pulling an arm corresponds to human-evaluating a model. By sampling adaptively based on the intermediate model rankings obtained on the samples so far, we can focus the annotation budget on the most competitive models. We prove the optimality of the proposed algorithms and show that it improves discrimination between top-performing models. This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.