Singular Value Scaling: Efficient Generative Model Compression via Pruned Weights Refinement
This work addresses inefficiencies in generative model compression for researchers and practitioners, offering a versatile method applicable to both GANs and Diffusion models, though it is incremental as it builds on existing pruning techniques.
The paper tackles the problem of inefficient fine-tuning after pruning in generative models by introducing Singular Value Scaling (SVS), a technique that refines pruned weights to improve initialization, resulting in enhanced compression performance and faster fine-tuning across StyleGAN2, StyleGAN3, and DDPM models without extra training costs.
While pruning methods effectively maintain model performance without extra training costs, they often focus solely on preserving crucial connections, overlooking the impact of pruned weights on subsequent fine-tuning or distillation, leading to inefficiencies. Moreover, most compression techniques for generative models have been developed primarily for GANs, tailored to specific architectures like StyleGAN, and research into compressing Diffusion models has just begun. Even more, these methods are often applicable only to GANs or Diffusion models, highlighting the need for approaches that work across both model types. In this paper, we introduce Singular Value Scaling (SVS), a versatile technique for refining pruned weights, applicable to both model types. Our analysis reveals that pruned weights often exhibit dominant singular vectors, hindering fine-tuning efficiency and leading to suboptimal performance compared to random initialization. Our method enhances weight initialization by minimizing the disparities between singular values of pruned weights, thereby improving the fine-tuning process. This approach not only guides the compressed model toward superior solutions but also significantly speeds up fine-tuning. Extensive experiments on StyleGAN2, StyleGAN3 and DDPM demonstrate that SVS improves compression performance across model types without additional training costs. Our code is available at: https://github.com/LAIT-CVLab/Singular-Value-Scaling.