Open, Closed, or Small Language Models for Text Classification?
This work provides practical guidance for model selection in text classification, though it is incremental as it compares existing models without introducing new methods.
The study evaluated open-source, closed-source, and small language models on text classification tasks, finding that fine-tuned open-source models can rival closed ones, while supervised smaller models like RoBERTa often match or exceed generative LLMs, but closed models excel in tasks requiring high generalizability.
Recent advancements in large language models have demonstrated remarkable capabilities across various NLP tasks. But many questions remain, including whether open-source models match closed ones, why these models excel or struggle with certain tasks, and what types of practical procedures can improve performance. We address these questions in the context of classification by evaluating three classes of models using eight datasets across three distinct tasks: named entity recognition, political party prediction, and misinformation detection. While larger LLMs often lead to improved performance, open-source models can rival their closed-source counterparts by fine-tuning. Moreover, supervised smaller models, like RoBERTa, can achieve similar or even greater performance in many datasets compared to generative LLMs. On the other hand, closed models maintain an advantage in hard tasks that demand the most generalizability. This study underscores the importance of model selection based on task requirements