APEX4: Efficient Pure W4A4 LLM Inference via Intra-SM Compute Rebalancing
For LLM inference practitioners, this work provides a systematic understanding and practical solution to make pure W4A4 quantization efficient across diverse GPU architectures.
W4A4 quantization is often avoided due to group dequantization overhead on CUDA Cores. The authors identify the Tensor Cores to CUDA Cores throughput ratio (ρ) as the key hardware indicator, showing that W4A4 viability is platform-dependent. They build APEX4, which co-designs pure INT4 GEMM kernels with ρ-aware granularity adaptation, achieving up to 2.09× end-to-end speedup on GPUs with favorable ρ and recovering A100 to 1.20–1.40×.
W4A4 quantization promises full utilization of INT4 Tensor Cores, yet group dequantization overhead on CUDA Cores has driven existing systems to mixed-precision fallbacks. We present the first systematic study of how intra-SM compute balance governs this bottleneck. Through controlled benchmarks across four GPUs from Ampere and Ada architectures, we identify the Tensor Cores to CUDA Cores throughput ratio ($ρ$) as the primary hardware indicator: the W4A4-g128 kernel yields $2.0$--$2.5\times$ speedup on RTX~3090 ($ρ=16$) yet degrades to $0.43$--$0.47\times$ on A100 ($ρ=64$) in compute-bond scenarios, establishing W4A4 viability as platform-dependent rather than universally infeasible. Guided by this finding, we build \textbf{APEX4}, which co-designs pure INT4 GEMM kernels with $ρ$-aware granularity adaptation to mitigate the CUDA Cores dequantization bottleneck. APEX4 achieves perplexity within 0.63 of FP16 on LLaMA-2-70B and outperforms W4Ax Atom-g128 by 4.0\%--4.4\% in zero-shot accuracy. Deployed as a drop-in replacement in unmodified vLLM, it delivers up to $1.66\times$ end-to-end speedup on L40S ($ρ=8$), and $1.78\times$ on RTX~3090 ($ρ=16$), $2.09\times$ on A40 ($ρ=16$), while recovering A100 ($ρ=64$) to $1.20$--$1.40\times$ via the mixed-granularity mode.