Computational Bottlenecks of Training Small-scale Large Language Models
This work addresses efficiency challenges for low-resource AI research institutes, though it is incremental as it applies existing methods to new data.
The study investigated computational bottlenecks in training small-scale large language models (up to 2B parameters) by analyzing hyperparameters like GPU type and batch size, finding optimizations for metrics such as loss per dollar and tokens per second.
While large language models (LLMs) dominate the AI landscape, Small-scale large Language Models (SLMs) are gaining attention due to cost and efficiency demands from consumers. However, there is limited research on the training behavior and computational requirements of SLMs. In this study, we explore the computational bottlenecks of training SLMs (up to 2B parameters) by examining the effects of various hyperparameters and configurations, including GPU type, batch size, model size, communication protocol, attention type, and the number of GPUs. We assess these factors on popular cloud services using metrics such as loss per dollar and tokens per second. Our findings aim to support the broader adoption and optimization of language model training for low-resource AI research institutes.