not-so-BigGAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
This reduces the resource barrier for high-fidelity image generation, making it more accessible to researchers with limited compute, though it is an incremental improvement over existing methods.
The paper tackles the high computational cost of training state-of-the-art high-resolution image generation models by proposing a two-step framework that generates images in low-frequency wavelet bands and super-resolves them, achieving an FID of 10.59 on ImageNet 512x512 with half the compute of BigGAN.
State-of-the-art models for high-resolution image generation, such as BigGAN and VQVAE-2, require an incredible amount of compute resources and/or time (512 TPU-v3 cores) to train, putting them out of reach for the larger research community. On the other hand, GAN-based image super-resolution models, such as ESRGAN, can not only upscale images to high dimensions, but also are efficient to train. In this paper, we present not-so-big-GAN (nsb-GAN), a simple yet cost-effective two-step training framework for deep generative models (DGMs) of high-dimensional natural images. First, we generate images in low-frequency bands by training a sampler in the wavelet domain. Then, we super-resolve these images from the wavelet domain back to the pixel-space with our novel wavelet super-resolution decoder network. Wavelet-based down-sampling method preserves more structural information than pixel-based methods, leading to significantly better generative quality of the low-resolution sampler (e.g., 64x64). Since the sampler and decoder can be trained in parallel and operate on much lower dimensional spaces than end-to-end models, the training cost is substantially reduced. On ImageNet 512x512, our model achieves a Fréchet Inception Distance (FID) of 10.59 -- beating the baseline BigGAN model -- at half the compute (256 TPU-v3 cores).