Large Language Model Selection with Limited Annotations
For practitioners needing to choose among many LLMs for a task, SELECT-LLM dramatically reduces the annotation cost of evaluation, making model selection more efficient.
SELECT-LLM is the first framework for active model selection of LLMs, using expected information gain from pairwise output similarities to identify the best LLM with minimal annotations. It achieves annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection across 23 datasets and 156 models.
Choosing a Large Language Model (LLM) for a given task requires comparing many strong candidates, yet standard evaluation relies on costly annotations over fixed evaluation sets. To address this challenge, we develop SELECT-LLM, the first framework for active model selection of LLMs. SELECT-LLM aims to find a small set of queries whose annotations are most informative for identifying the best LLM for a given task. To this end, we introduce a query selection rule based on expected information gain, computed from pairwise similarities between candidate model outputs. Because this rule only uses generated model responses, SELECT-LLM can be applied across candidate models without assumptions about their architecture or access to model weights. This makes it suitable for both open-weight and black-box LLMs. We evaluate SELECT-LLM across 23 datasets, 156 evaluated models, diverse task families, and multiple text evaluation metrics. Across all experiments, SELECT-LLM improves over the strongest baseline in every setting, with annotation cost reductions up to 81.8% for best model selection and up to 84.78% for near-best model selection.