Improving In-Context Learning with Small Language Model Ensembles
This addresses the need for efficient domain specialization methods in large language models, offering a cheap and effective solution for practitioners, though it appears incremental as it builds on existing in-context learning and ensemble techniques.
The paper tackles the problem of limited performance of large language models on domain-specific tasks by introducing Ensemble SuperICL, which enhances in-context learning using multiple fine-tuned small language models, achieving state-of-the-art results on natural language understanding benchmarks and superior accuracy in a medical-domain labelling task compared to baselines.
Large language models (LLMs) have shown impressive capabilities across various tasks, but their performance on domain-specific tasks remains limited. While methods like retrieval augmented generation and fine-tuning can help to address this, they require significant resources. In-context learning (ICL) is a cheap and efficient alternative but cannot match the accuracies of advanced methods. We present Ensemble SuperICL, a novel approach that enhances ICL by leveraging the expertise of multiple fine-tuned small language models (SLMs). Ensemble SuperICL achieves state of the art (SoTA) results on several natural language understanding benchmarks. Additionally, we test it on a medical-domain labelling task and showcase its practicality by using off-the-shelf SLMs fine-tuned on a general language task, achieving superior accuracy in large-scale data labelling compared to all baselines. Finally, we conduct an ablation study and sensitivity analyses to elucidate the underlying mechanism of Ensemble SuperICL. Our research contributes to the growing demand for efficient domain specialisation methods in LLMs, offering a cheap and effective method for practitioners.