TinyGAN: Distilling BigGAN for Conditional Image Generation
This work addresses the deployment challenge for conditional image generation on devices with limited resources, representing an incremental improvement in model compression.
The paper tackles the problem of deploying large-scale GANs like BigGAN on resource-constrained devices by proposing a black-box knowledge distillation framework to compress the model, achieving competitive performance with a generator that has 16 times fewer parameters.
Generative Adversarial Networks (GANs) have become a powerful approach for generative image modeling. However, GANs are notorious for their training instability, especially on large-scale, complex datasets. While the recent work of BigGAN has significantly improved the quality of image generation on ImageNet, it requires a huge model, making it hard to deploy on resource-constrained devices. To reduce the model size, we propose a black-box knowledge distillation framework for compressing GANs, which highlights a stable and efficient training process. Given BigGAN as the teacher network, we manage to train a much smaller student network to mimic its functionality, achieving competitive performance on Inception and FID scores with the generator having $16\times$ fewer parameters.