Scaling Up Active Testing to Large Language Models
This addresses the computational bottleneck for label-efficient evaluation of large language models, though it appears incremental as it builds on existing active testing methods.
The paper tackles the problem of scaling active testing to large language models by showing that surrogate models can be constructed cheaply using in-context learning, enabling more effective evaluation with less data than standard practices.
Active testing enables label-efficient evaluation of models through careful data acquisition. However, its significant computational costs have previously undermined its use for large models. We show how it can be successfully scaled up to the evaluation of large language models (LLMs). In particular we show that the surrogate model used to guide data acquisition can be constructed cheaply using in-context learning, does not require updating within an active-testing loop, and can be smaller than the target model. We even find we can make good data-acquisition decisions without computing predictions with the target model and further introduce a single-run error estimator to asses how well active testing is working on the fly. We find that our approach is able to more effectively evaluate LLM performance with less data than current standard practices.