The Race to Efficiency: A New Perspective on AI Scaling Laws
This work addresses the challenge of sustaining AI progress for researchers and industry by balancing compute investments with efficiency improvements, though it is incremental as it builds on existing scaling laws.
The paper tackles the problem of escalating costs and diminishing returns in training large-scale AI models by introducing a time- and efficiency-aware framework that extends classical scaling laws, showing that without ongoing efficiency gains, advanced performance could require millennia of training, but near-exponential progress remains possible if efficiency improvements parallel Moore's Law.
As large-scale AI models expand, training becomes costlier and sustaining progress grows harder. Classical scaling laws (e.g., Kaplan et al. (2020), Hoffmann et al. (2022)) predict training loss from a static compute budget yet neglect time and efficiency, prompting the question: how can we balance ballooning GPU fleets with rapidly improving hardware and algorithms? We introduce the relative-loss equation, a time- and efficiency-aware framework that extends classical AI scaling laws. Our model shows that, without ongoing efficiency gains, advanced performance could demand millennia of training or unrealistically large GPU fleets. However, near-exponential progress remains achievable if the "efficiency-doubling rate" parallels Moore's Law. By formalizing this race to efficiency, we offer a quantitative roadmap for balancing front-loaded GPU investments with incremental improvements across the AI stack. Empirical trends suggest that sustained efficiency gains can push AI scaling well into the coming decade, providing a new perspective on the diminishing returns inherent in classical scaling.