SmoothQuant

“SmoothQuant can incur a significant accuracy degradation of up to 15 points on Llama2-70B in W4A4 scenarios”

“smoothing-based methods often transfer the burden of quantization from activations to weights without eliminating it.”

“existing PTQ approaches for LLMs minimize the layer-wise reconstruction loss while treating all tokens uniformly lin2023awq,frantar2022gptq,li2025gptqv2, without accounting for token-level informativeness or importance. Such a token-agnostic design inevitably biases the quantized model toward dominant but redundant visual features”

“However, GPTQ and SmoothQuant~(SQ), which are strong PTQ methods for pure LLMs, do not reliably improve performance in this multimodal setting.”

“While these approaches effectively stabilize numerical distributions of language and 2D vision models, they generalize poorly to 3D geometry networks such as VGGT.”

“However, these methods cannot effectively quantize the LLMs to 4-bit weights and activations.”

“Such methods provide useful insight that token distributions are highly non-uniform, yet their main lever is changing the bit-width.”

“SmoothQuant~xiao2023smoothquant smooths the distribution of LayerNorm activations before quantization. Unfortunately, when performing low-bit (e.g., 4-bit) quantization, these schemes tend to underperform significantly”

“SmoothQuant suffers from non-negligible performance degradation for other open-source models such as Llama and Llama $2$ with a $8$-bit per-tensor static activation quantization scheme”

“we observe notable degradation on more complex tasks such as math reasoning and code generation”

“However, these methods operate primarily along the feature dimension and ignore correlations across the sequence dimension.”

“Though alleviating the outlier issues, the asymmetry in activations is still challenging to symmetric quantization.”