Communication Compression for Tensor Parallel LLM Inference
This work addresses inference speed bottlenecks for users deploying large language models on multiple accelerators, representing an incremental improvement in communication efficiency.
The paper tackles the problem of high latency in tensor parallel LLM inference by compressing inter-accelerator communication using fine-grained quantization, achieving up to a 2x reduction in time-to-first-token with minimal performance degradation.
Large Language Models (LLMs) have pushed the frontier of artificial intelligence but are comprised of hundreds of billions of parameters and operations. For faster inference latency, LLMs are deployed on multiple hardware accelerators through various Model Parallelism strategies. Our paper looks into the details on one such strategy - Tensor Parallel - and proposes to reduce latency by compressing inter-accelerator communication. We leverage fine grained quantization techniques to compress selected activations by 3.5 - 4.5x. Our proposed method leads up to 2x reduction of time-to-first-token (TTFT) with negligible model performance degradation.