Scaling Laws Are Unreliable for Downstream Tasks: A Reality Check
This work highlights a critical limitation for researchers and practitioners relying on scaling laws to forecast AI model performance, revealing it as incremental by building on prior debates about scaling trends.
The paper tackles the problem of predicting downstream task performance from scaling laws, finding that predictable scaling occurs only 39% of the time and is highly sensitive to experimental changes.
Downstream scaling laws aim to predict task performance at larger scales from the model's performance at smaller scales. Whether such prediction should be possible is unclear: some works discover clear linear scaling trends after simple transformations of the performance metric, whereas others point out fundamental challenges to downstream scaling laws, such as emergence and inverse scaling. In this work, we conduct a meta-analysis of existing data on downstream scaling laws, and we find that predictable scaling only occurs in a minority of cases: 39% of the time. Moreover, seemingly benign changes to the experimental setting can completely change the scaling behavior. Our analysis underscores the need to understand the conditions under which scaling laws succeed. To accurately model the relationship between pretraining loss and task performance, we must embrace the cases in which scaling behavior deviates from linear trends.