More Than Bits: Multi-Envelope Double Binary Factorization for Extreme Quantization
This work addresses a specific bottleneck in efficient LLM deployment for resource-constrained environments, representing an incremental improvement over existing methods.
The paper tackles the performance saturation problem in extreme low-bit quantization of large language models (LLMs) by proposing Multi-envelope DBF (MDBF), which enhances perplexity and zero-shot accuracy over previous binary formats at matched bits per weight.
For extreme low-bit quantization of large language models (LLMs), Double Binary Factorization (DBF) is attractive as it enables efficient inference without sacrificing accuracy. However, the scaling parameters of DBF are too restrictive; after factoring out signs, all rank components share the same magnitude profile, resulting in performance saturation. We propose Multi-envelope DBF (MDBF), which retains a shared pair of 1-bit sign bases but replaces the single envelope with a rank-$l$ envelope. By sharing sign matrices among envelope components, MDBF effectively maintains a binary carrier and utilizes the limited memory budget for magnitude expressiveness. We also introduce a closed-form initialization and an alternating refinement method to optimize MDBF. Across the LLaMA and Qwen families, MDBF enhances perplexity and zero-shot accuracy over previous binary formats at matched bits per weight while preserving the same deployment-friendly inference primitive.