Understanding LLMs: A Comprehensive Overview from Training to Inference
It provides a comprehensive overview for researchers and practitioners interested in cost-efficient LLM development, but it is incremental as it synthesizes existing knowledge without introducing novel methods.
This paper reviews the evolution of training techniques and inference deployment technologies for Large Language Models (LLMs), covering aspects like data preprocessing, model compression, and future trends, without presenting new experimental results or specific numerical improvements.
The introduction of ChatGPT has led to a significant increase in the utilization of Large Language Models (LLMs) for addressing downstream tasks. There's an increasing focus on cost-efficient training and deployment within this context. Low-cost training and deployment of LLMs represent the future development trend. This paper reviews the evolution of large language model training techniques and inference deployment technologies aligned with this emerging trend. The discussion on training includes various aspects, including data preprocessing, training architecture, pre-training tasks, parallel training, and relevant content related to model fine-tuning. On the inference side, the paper covers topics such as model compression, parallel computation, memory scheduling, and structural optimization. It also explores LLMs' utilization and provides insights into their future development.