Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM
This project addresses the need for open-source, high-quality language models, particularly for the Chinese language, which can benefit researchers and practitioners in the field of natural language processing.
The authors developed Steel-LLM, a 1-billion-parameter Chinese-centric language model, which demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. The model was trained on a large-scale dataset with limited computational resources.
Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM.