HARP: Hadamard-Preconditioned Adaptive Rotation Processor for Extreme LLM Quantization
For practitioners deploying large language models under memory constraints, HARP offers a practical method to improve extreme low-bit quantization accuracy without sacrificing inference speed.
HARP introduces a learnable structured rotation processor that adapts the quantization basis per layer, improving perplexity and zero-shot accuracy over fixed Hadamard transforms in extreme low-bit LLM quantization while maintaining high deployment efficiency (128 tok/s vs 61 tok/s for FP16).
Post-training quantization (PTQ) is essential for deploying LLMs under memory and bandwidth constraints. However, extreme low-bit quantization remains highly sensitive to activation outliers and anisotropic weight curvature. Existing incoherence-based PTQ methods mitigate this issue with fixed randomized Hadamard transforms (RHTs), which improve quantization robustness but cannot adapt the rotated basis to the layer, calibration distribution, or quantizer. We introduce HARP (Hadamard-preconditioned Adaptive Rotation Processor), a learnable structured two-sided orthogonal processor that replaces fixed Hadamard mixing while preserving exact full-precision equivalence. HARP represents each rotation as a product of sparse butterfly-like block-orthogonal stages, supports non-power-of-two dimensions via Mixed-Radix schedules, and initializes to the RHT processor up to a fixed permutation. Fitted only on calibration data, HARP adapts the quantization basis to each layer and backend. Across 2-4 bit settings on models ranging from 1B to 70B parameters, HARP improves perplexity and zero-shot accuracy over fixed RHT. Importantly, HARP preserves deployment efficiency, reaching 128 tok/s versus 61 tok/s for FP16.