What is the Role of Small Models in the LLM Era: A Survey
It tackles the problem of high computational costs and limited accessibility of large models for academic researchers and businesses, offering insights for practitioners, but it is incremental as a survey.
This survey examines the role of small models in the era of large language models, addressing their underestimated significance and exploring their relationship through collaboration and competition to promote more efficient computational resource use.
Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models