LEXI: Lossless Exponent Coding for Efficient Inter-Chiplet Communication in Hybrid LLMs
This addresses the issue of high communication costs in hybrid LLM chiplet architectures, offering a domain-specific optimization for efficient AI hardware.
The paper tackles the problem of data movement overheads increasing inference latency in large language models by proposing LEXI, a lossless exponent compression scheme based on Huffman coding, which reduces inter-chiplet communication latency by 33-45% and end-to-end inference latency by 30-35% on models like Jamba, Zamba, and Qwen.
Data movement overheads increase the inference latency of state-of-the-art large language models (LLMs). These models commonly use the bfloat16 (BF16) format for stable training. Floating-point standards allocate eight bits to the exponent, but our profiling reveals that exponent streams exhibit fewer than 3 bits Shannon entropy, indicating high inherent compressibility. To exploit this potential, we propose LEXI, a novel lossless exponent compression scheme based on Huffman coding. LEXI compresses activations and caches on the fly while storing compressed weights for just-in-time decompression near compute, without sacrificing system throughput and model accuracy. The codecs at the ingress and egress ports of network-on-chip routers sustain the maximum link bandwidth via multi-lane LUT decoders, incurring only 0.09 percent area and energy overheads with GF 22 nm technology. LEXI reduces inter-chiplet communication and end-to-end inference latencies by 33-45 percent and 30-35 percent on modern Jamba, Zamba, and Qwen LLMs implemented on a homogeneous chiplet architecture.