Karthikeyan Sankaralingam

Karthikeyan Sankaralingam

The end of Moore's Law and Dennard scaling has fundamentally changed the economics of computer architecture. With transistor scaling delivering diminishing returns, architectural innovation is now the primary - and perhaps only - remaining lever for performance improvement. However, we argue that human-driven architecture research is fundamentally ill-suited for this new era. The architectural design space is vast (effectively infinite for practical purposes), yet human teams explore perhaps 50-100 designs per generation, sampling less than 0.001% of possibilities. This approach worked during the abundance era when Moore's Law provided a rising tide that lifted all designs. In the current scarcity paradigm, where every architecture must deliver 2X performance improvements using essentially the same transistor budget, systematic exploration becomes critical. We propose a concrete alternative: automated idea factories that generate and evaluate thousands of candidate architectures weekly through multi-tiered evaluation pipelines, learning from deployed telemetry data in a continuous feedback loop. Early results suggest that such systems can compress architectural design cycles from double-digit months to single-digit weeks by exploring orders of magnitude more candidates than any human team, and do it much faster. We predict that within 2 years, purely human-driven architecture research will be as obsolete as human chess players competing against engines.

Yanxin Zhang, Ian McDougall, Junnan Li et al.

For over a decade, processor design has focused on implementing sophisticated policies for various components of the out-of-order pipeline, including cache replacement and prefetching. The prevailing design philosophy has been to build processors with a single, static selection of policies across these different mechanisms. This paper investigates a fundamental question: do different workloads, or even different execution phases within the same workload, benefit from different policy combinations? We present a comprehensive analysis exploring whether a hypothetical processor capable of dynamically selecting from multiple policies could significantly outperform traditional static-policy processors. Using ChampSim-based simulation across 49 benchmarks segmented into 490 execution phases of 20M instructions each, we evaluate performance across multiple policy combinations for cache replacement and prefetching. Our findings reveal that significant performance headroom exists: the best static policy achieves optimal performance for only 19.18\% of execution phases and incurs a mean IPC loss of 1.54\% compared to an oracle. Moreover, 85 phases (17.35\%), spanning 14 of the 49 applications, exhibit more than 2.5\% IPC loss relative to the oracle. Furthermore, we demonstrate that a processor capable of dynamically switching between two carefully chosen policies can achieve a 13.6$\times$ reduction in mean IPC loss (from 1.54\% to 0.11\%) and match oracle performance 52.65\% of the time. These results suggest that dynamic policy selection represents a promising avenue for unlocking single-thread performance improvements that have become increasingly difficult to achieve.

Zhanpeng Zeng, Michael Davies, Pranav Pulijala et al.

While GPU clusters are the de facto choice for training large deep neural network (DNN) models today, several reasons including ease of workflow, security and cost have led to efforts investigating whether CPUs may be viable for inference in routine use in many sectors of the industry. But the imbalance between the compute capabilities of GPUs and CPUs is huge. Motivated by these considerations, we study a module which is a workhorse within modern DNN architectures, GEMM based Feed Forward Networks (FFNs), and assess the extent to which it can be made compute- (or FLOP-) lite. Specifically, we propose an alternative formulation (we call it LookupFFN) to GEMM based FFNs inspired by the recent studies of using Locality Sensitive Hashing (LSH) to approximate FFNs. Our formulation recasts most essential operations as a memory look-up, leveraging the trade-off between the two resources on any platform: compute and memory (since CPUs offer it in abundance). For RoBERTa language model pretraining, our formulation achieves similar performance compared to GEMM based FFNs, while dramatically reducing the required FLOP. Our development is complemented with a detailed hardware profiling of strategies that will maximize efficiency -- not just on contemporary hardware but on products that will be offered in the near/medium term future. Code is avaiable at \url{https://github.com/mlpen/LookupFFN}.

Zhanpeng Zeng, Karthikeyan Sankaralingam, Vikas Singh

GEneral Matrix Multiply (GEMM) is a central operation in deep learning and corresponds to the largest chunk of the compute footprint. Therefore, improving its efficiency is an active topic of ongoing research. A popular strategy is the use of low bit-width integers to approximate the original entries in a matrix. This allows efficiency gains, but often requires sophisticated techniques to control the rounding error incurred. In this work, we first verify/check that when the low bit-width restriction is removed, for a variety of Transformer-based models, whether integers are sufficient for all GEMMs need -- for {\em both} training and inference stages, and can achieve parity with floating point counterparts. No sophisticated techniques are needed. We find that while a large majority of entries in matrices (encountered in such models) can be easily represented by {\em low} bit-width integers, the existence of a few heavy hitter entries make it difficult to achieve efficiency gains via the exclusive use of low bit-width GEMMs alone. To address this issue, we develop a simple algorithm, Integer Matrix Unpacking (IM-Unpack), to {\em unpack} a matrix with large integer entries into a larger matrix whose entries all lie within the representable range of arbitrarily low bit-width integers. This allows {\em equivalence} with the original GEMM, i.e., the exact result can be obtained using purely low bit-width integer GEMMs. This comes at the cost of additional operations -- we show that for many popular models, this overhead is quite small.