The Non-Determinism of Small LLMs: Evidence of Low Answer Consistency in Repetition Trials of Standard Multiple-Choice Benchmarks
This work addresses the problem of unreliable outputs from small LLMs for users in benchmarking and practical applications, highlighting an incremental analysis of consistency trade-offs.
The study investigated the consistency of small LLMs (2B-8B parameters) in answering the same questions multiple times, finding that they typically achieve 50%-80% consistency at low temperatures, with accuracy correlating reasonably with overall accuracy, while medium-sized models (50B-80B) showed much higher consistency.
This work explores the consistency of small LLMs (2B-8B parameters) in answering multiple times the same question. We present a study on known, open-source LLMs responding to 10 repetitions of questions from the multiple-choice benchmarks MMLU-Redux and MedQA, considering different inference temperatures, small vs. medium models (50B-80B), finetuned vs. base models, and other parameters. We also look into the effects of requiring multi-trial answer consistency on accuracy and the trade-offs involved in deciding which model best provides both of them. To support those studies, we propose some new analytical and graphical tools. Results show that the number of questions which can be answered consistently vary considerably among models but are typically in the 50%-80% range for small models at low inference temperatures. Also, accuracy among consistent answers seems to reasonably correlate with overall accuracy. Results for medium-sized models seem to indicate much higher levels of answer consistency.