Shift-and-Sum Quantization for Visual Autoregressive Models
Enables efficient deployment of VAR models for practitioners needing low-bit quantization without retraining.
This paper tackles post-training quantization for visual autoregressive models (VAR), addressing large reconstruction errors in attention-value products and calibration data mismatch. The proposed shift-and-sum quantization and resampling strategy achieve state-of-the-art PTQ results on class-conditional image generation, inpainting, outpainting, and editing.
Post-training quantization (PTQ) enables efficient deployment of deep networks using a small set of data. Its application to visual autoregressive models (VAR), however, remains relatively unexplored. We identify two key challenges for applying PTQ to VAR: (i) large reconstruction errors in attention-value products, especially at coarse scales where high attention scores occur more frequently; and (ii) a discrepancy between the sampling frequencies of codebook entries and their predicted probabilities due to limited calibration data. To address these challenges, we propose a PTQ framework tailored for VAR. First, we introduce a shift-and-sum quantization method that reduces reconstruction errors by aggregating quantized results from symmetrically shifted duplicates of value tokens. Second, we present a resampling strategy for calibration data that aligns sampling frequencies of codebook entries with their predicted probabilities. Experiments on class-conditional image generation, inpainting, outpainting, and class-conditional editing show consistent improvements across VAR architectures, establishing a new state of the art in PTQ for VAR.