Using tournaments to calculate AUROC for zero-shot classification with LLMs
This work addresses a methodological gap for researchers evaluating LLMs in zero-shot classification, though it is incremental as it builds on existing pairwise comparison and Elo rating techniques.
The paper tackles the problem of fairly comparing large language models (LLMs) to supervised classifiers in zero-shot binary classification by converting tasks into pairwise comparisons and using the Elo rating system to score instances, showing that their proposed scheduling algorithm improves classification performance and provides more information than traditional methods.
Large language models perform surprisingly well on many zero-shot classification tasks, but are difficult to fairly compare to supervised classifiers due to the lack of a modifiable decision boundary. In this work, we propose and evaluate a method that converts binary classification tasks into pairwise comparison tasks, obtaining relative rankings from LLMs. Repeated pairwise comparisons can be used to score instances using the Elo rating system (used in chess and other competitions), inducing a confidence ordering over instances in a dataset. We evaluate scheduling algorithms for their ability to minimize comparisons, and show that our proposed algorithm leads to improved classification performance, while also providing more information than traditional zero-shot classification.