LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws
This work addresses the problem of optimizing LLM performance for practitioners by emphasizing data curation over architectural choices, though it is incremental in refining existing scaling law insights.
The study investigated factors influencing loss-to-loss scaling laws in large language models, finding that pretraining data and tokenizer primarily determine scaling trends, while model size, hyperparameters, and architecture have limited impact.
Scaling laws guide the development of large language models (LLMs) by offering estimates for the optimal balance of model size, tokens, and compute. More recently, loss-to-loss scaling laws that relate losses across pretraining datasets and downstream tasks have emerged as a powerful tool for understanding and improving LLM performance. In this work, we investigate which factors most strongly influence loss-to-loss scaling. Our experiments reveal that the pretraining data and tokenizer determine the scaling trend. In contrast, model size, optimization hyperparameters, and even significant architectural differences, such as between transformer-based models like Llama and state-space models like Mamba, have limited impact. Consequently, practitioners should carefully curate suitable pretraining datasets for optimal downstream performance, while architectures and other settings can be freely optimized for training efficiency.