Scaling Laws for Code: A More Data-Hungry Regime
This work addresses the need for efficient training of code large language models, which is crucial for software engineering, but it is incremental as it adapts existing scaling laws to a new domain.
The study tackled the problem of whether scaling laws for natural language apply to code by conducting a large-scale empirical analysis with 117 runs, revealing that code is more data-hungry, requiring a higher data-to-parameter ratio than natural language, and that the Farseer law offers greater accuracy.
Code Large Language Models (LLMs) are revolutionizing software engineering. However, scaling laws that guide the efficient training are predominantly analyzed on Natural Language (NL). Given the fundamental differences like strict syntax between code and NL, it is unclear whether these laws are directly applicable to code. To address this gap, we conduct the first large-scale empirical study of scaling laws for code, comprising 117 experimental runs with model sizes from 0.2B to 3.8B and training tokens from 2B to 128B. We fit the Chinchilla law and the Farsser law. First, the results show that the more expressive Farseer law offers greater accuracy. Second, the analysis reveals that Code LLMs scale effectively with model size. Crucially, code represents a more data-hungry regime, requiring a substantially higher data-to-parameter ratio than NL. Finally, two additional sets of experiments on code-NL mixtures show that NL benefits resource-constrained scenarios, but becomes a detriment at higher compute budgets.