Compute Requirements for Algorithmic Innovation in Frontier AI Models
This addresses the problem of understanding and managing compute resources for AI innovation, particularly for policymakers and researchers, but is incremental as it builds on existing data and analysis.
The paper investigates the compute requirements for developing algorithmic innovations in large language models, finding that innovations using significant resources double in requirements each year, and suggests that compute caps alone are unlikely to dramatically slow AI algorithmic progress, with stringent caps still allowing half of the cataloged innovations.
Algorithmic innovation in the pretraining of large language models has driven a massive reduction in the total compute required to reach a given level of capability. In this paper we empirically investigate the compute requirements for developing algorithmic innovations. We catalog 36 pre-training algorithmic innovations used in Llama 3 and DeepSeek-V3. For each innovation we estimate both the total FLOP used in development and the FLOP/s of the hardware utilized. Innovations using significant resources double in their requirements each year. We then use this dataset to investigate the effect of compute caps on innovation. Our analysis suggests that compute caps alone are unlikely to dramatically slow AI algorithmic progress. Even stringent compute caps -- such as capping total operations to the compute used to train GPT-2 or capping hardware capacity to 8 H100 GPUs -- could still have allowed for half of the cataloged innovations.