Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers
This enables privacy, cost, and explainability benefits for users in commercial and democratized AI applications, though it is incremental as it applies an existing method to a new context.
The paper tackles the problem of achieving high performance in few-shot classification without relying on large commercial language models, showing that logistic regression on small LLM embeddings matches or exceeds large LLM performance on 17 tasks with 2-4 classes, using only tens of labeled examples.
For simple classification tasks, we show that users can benefit from the advantages of using small, local, generative language models instead of large commercial models without a trade-off in performance or introducing extra labelling costs. These advantages, including those around privacy, availability, cost, and explainability, are important both in commercial applications and in the broader democratisation of AI. Through experiments on 17 sentence classification tasks (2-4 classes), we show that penalised logistic regression on the embeddings from a small LLM equals (and usually betters) the performance of a large LLM in the "tens-of-shot" regime. This requires no more labelled instances than are needed to validate the performance of the large LLM. Finally, we extract stable and sensible explanations for classification decisions.