CarbonScaling: Extending Neural Scaling Laws for Carbon Footprint in Large Language Models
This addresses the environmental impact of LLM training for AI researchers and practitioners, though it is incremental as it builds on existing neural scaling laws.
The paper tackles the problem of carbon emissions scaling with large language model (LLM) size by extending neural scaling laws to incorporate operational and embodied carbon, showing that real-world inefficiencies significantly increase the scaling factor and that hardware scaling has diminishing returns for extremely large LLMs.
Neural scaling laws have driven the development of increasingly large language models (LLMs) by linking accuracy improvements to growth in parameter count, dataset size, and compute. However, these laws overlook the carbon emissions that scale exponentially with LLM size. This paper presents \textit{CarbonScaling}, an analytical framework that extends neural scaling laws to incorporate both operational and embodied carbon in LLM training. By integrating models for neural scaling, GPU hardware evolution, parallelism optimization, and carbon estimation, \textit{CarbonScaling} quantitatively connects model accuracy to carbon footprint. Results show that while a power-law relationship between accuracy and carbon holds, real-world inefficiencies significantly increase the scaling factor. Hardware technology scaling reduces carbon emissions for small to mid-sized models, but offers diminishing returns for extremely large LLMs due to communication overhead and underutilized GPUs. Training optimizations-especially aggressive critical batch size scaling-help alleviate this inefficiency. \textit{CarbonScaling} offers key insights for training more sustainable and carbon-efficient LLMs.