Fast NF4 Dequantization Kernels for Large Language Model Inference
For practitioners deploying large LLMs on limited GPU memory, this work provides a practical, plug-and-play optimization that improves inference speed without changing the quantization format or ecosystem compatibility.
This paper addresses the dequantization bottleneck in NF4-quantized LLM inference on NVIDIA GPUs. By optimizing memory hierarchy usage with a lightweight shared memory approach, they achieve 2.0–2.2× kernel speedup and up to 1.54× end-to-end improvement over the BitsAndBytes baseline across three large models.
Large language models (LLMs) have grown beyond the memory capacity of single GPU devices, necessitating quantization techniques for practical deployment. While NF4 (4-bit NormalFloat) quantization enables 4$\times$ memory reduction, inference on current NVIDIA GPUs (e.g., Ampere A100) requires expensive dequantization back to FP16 format, creating a critical performance bottleneck. This paper presents a lightweight shared memory optimization that addresses this gap through principled memory hierarchy exploitation while maintaining full ecosystem compatibility. We compare our technique against the open-source BitsAndBytes implementation, achieving 2.0--2.2$\times$ kernel speedup across three models (Gemma 27B, Qwen3 32B, and Llama3.3 70B) and up to 1.54$\times$ end-to-end improvement by leveraging the 12--15$\times$ latency advantage of shared memory over global memory access. Our optimization reduces instruction counts through simplified indexing logic while using only 64 bytes of shared memory per thread block, demonstrating that lightweight optimizations can deliver substantial performance gains with minimal engineering effort. This work provides a plug-and-play solution for the HuggingFace ecosystem that democratizes access to advanced models on existing GPU infrastructure.