Efficient Large Language Models: A Survey
It provides a comprehensive resource for researchers and practitioners working on efficiency challenges in LLMs, which is crucial for broader adoption and societal impact, but it is incremental as a survey rather than novel research.
This survey systematically reviews research on efficient large language models (LLMs) to address their high resource demands, organizing literature into model-centric, data-centric, and framework-centric categories and providing a GitHub repository for ongoing updates.
Large Language Models (LLMs) have demonstrated remarkable capabilities in important tasks such as natural language understanding and language generation, and thus have the potential to make a substantial impact on our society. Such capabilities, however, come with the considerable resources they demand, highlighting the strong need to develop effective techniques for addressing their efficiency challenges. In this survey, we provide a systematic and comprehensive review of efficient LLMs research. We organize the literature in a taxonomy consisting of three main categories, covering distinct yet interconnected efficient LLMs topics from model-centric, data-centric, and framework-centric perspective, respectively. We have also created a GitHub repository where we organize the papers featured in this survey at https://github.com/AIoT-MLSys-Lab/Efficient-LLMs-Survey. We will actively maintain the repository and incorporate new research as it emerges. We hope our survey can serve as a valuable resource to help researchers and practitioners gain a systematic understanding of efficient LLMs research and inspire them to contribute to this important and exciting field.