Active Model Selection for Large Language Models
This addresses the high annotation cost problem for researchers and practitioners evaluating LLMs, though it is an incremental improvement over existing evaluation methods.
The paper tackles the problem of efficiently selecting the best Large Language Model (LLM) for a task with limited annotations, introducing LLM SELECTOR which reduces annotation costs by up to 59.62% in experiments across 6 benchmarks with 151 LLMs.
We introduce LLM SELECTOR, the first framework for active model selection of Large Language Models (LLMs). Unlike prior evaluation and benchmarking approaches that rely on fully annotated datasets, LLM SELECTOR efficiently identifies the best LLM with limited annotations. In particular, for any given task, LLM SELECTOR adaptively selects a small set of queries to annotate that are most informative about the best model for the task. To further reduce annotation cost, we leverage a judge-based oracle annotation model. Through extensive experiments on 6 benchmarks with 151 LLMs, we show that LLM SELECTOR reduces annotation costs by up to 59.62% when selecting the best and near-best LLM for the task.