QuantFace: Low-Bit Post-Training Quantization for One-Step Diffusion Face Restoration
This work addresses the computational bottleneck for deploying diffusion-based face restoration on smartphones, representing an incremental improvement in model compression for a specific domain.
The authors tackled the problem of deploying diffusion models for face restoration on resource-constrained devices by proposing QuantFace, a low-bit post-training quantization method that reduces weights and activations to 4-6 bits, achieving significant advantages over existing methods in experiments.
Diffusion models have been achieving remarkable performance in face restoration. However, the heavy computations of diffusion models make it difficult to deploy them on devices like smartphones. In this work, we propose QuantFace, a novel low-bit quantization for one-step diffusion face restoration models, where the full-precision (\ie, 32-bit) weights and activations are quantized to 4$\sim$6-bit. We first analyze the data distribution within activations and find that they are highly variant. To preserve the original data information, we employ rotation-scaling channel balancing. Furthermore, we propose Quantization-Distillation Low-Rank Adaptation (QD-LoRA) that jointly optimizes for quantization and distillation performance. Finally, we propose an adaptive bit-width allocation strategy. We formulate such a strategy as an integer programming problem, which combines quantization error and perceptual metrics to find a satisfactory resource allocation. Extensive experiments on the synthetic and real-world datasets demonstrate the effectiveness of QuantFace under 6-bit and 4-bit. QuantFace achieves significant advantages over recent leading low-bit quantization methods for face restoration. The code is available at https://github.com/jiatongli2024/QuantFace.