Modernizing Amdahl's Law: How AI Scaling Laws Shape Computer Architecture
This addresses the problem of resource allocation in computer architecture for AI and heterogeneous systems, offering a theoretical framework to explain trends like increasing GPU programmability, but it is incremental as it builds on classical Amdahl's Law.
The paper reformulates Amdahl's Law for modern heterogeneous computer systems with scalable workloads, revealing a finite collapse threshold where specialization becomes suboptimal beyond a critical scalable fraction, leading to a phase transition rather than an asymptotic tail.
Classical Amdahl's Law assumes a fixed decomposition between serial and parallel work and homogeneous replication; historically, it bounds how much parallel speedup is attainable. Modern systems instead combine specialized accelerators with programmable compute, tensor datapaths, and evolving pipelines, while empirical scaling laws shift which stages absorb marginal compute. The central tension is therefore not the serial-versus-parallel split alone, but resource allocation across heterogeneous hardware, given efficiency differences, and workload structures that determine how effectively additional compute can be converted into value. We reformulate Amdahl's Law for modern heterogeneous systems with scalable workloads. The analysis yields a finite collapse threshold: beyond a critical scalable fraction, specialization becomes suboptimal for any efficiency advantage of specialized hardware over programmable compute, and optimal specialized investment falls to zero, a phase transition rather than an asymptotic tail. We use this framework to interpret increasing GPU programmability and why domain-specific AI accelerators have not displaced GPUs.