Generalizing Scaling Laws for Dense and Sparse Large Language Models
This work addresses the efficiency problem in training large language models for researchers and practitioners, but it is incremental as it builds on existing scaling laws.
The paper tackles the challenge of predicting model size and resource allocation for large language models by proposing a generalized scaling law that unifies dense and sparse architectures, demonstrating its effectiveness through evaluation and comparison with existing methods.
Over the past few years, the size of language models has grown exponentially, as has the computational cost to train these large models. This rapid growth has motivated researchers to develop new techniques aimed at enhancing the efficiency of the training process. Despite these advancements, optimally predicting the model size or allocating optimal resources remains a challenge. Several efforts have addressed the challenge by proposing different scaling laws, but almost all of them are architecture-specific (dense or sparse). In this work we revisit existing scaling laws and propose a generalized scaling law to provide a unified framework that is applicable to both dense and sparse large language models. We evaluate and compare our proposed scaling law with existing scaling laws to demonstrate its effectiveness.