GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks
This work addresses training instability in GANs for image generation, offering a novel normalization technique that is incremental but provides specific gains over existing methods.
The paper tackled the problem of stabilizing GAN training by introducing Gradient Normalization (GraN), a method that enforces a piecewise Lipschitz constraint in discriminators, leading to improved image generation performance across multiple datasets like CIFAR-10/100 and CelebA.
Modern generative adversarial networks (GANs) predominantly use piecewise linear activation functions in discriminators (or critics), including ReLU and LeakyReLU. Such models learn piecewise linear mappings, where each piece handles a subset of the input space, and the gradients per subset are piecewise constant. Under such a class of discriminator (or critic) functions, we present Gradient Normalization (GraN), a novel input-dependent normalization method, which guarantees a piecewise K-Lipschitz constraint in the input space. In contrast to spectral normalization, GraN does not constrain processing at the individual network layers, and, unlike gradient penalties, strictly enforces a piecewise Lipschitz constraint almost everywhere. Empirically, we demonstrate improved image generation performance across multiple datasets (incl. CIFAR-10/100, STL-10, LSUN bedrooms, and CelebA), GAN loss functions, and metrics. Further, we analyze altering the often untuned Lipschitz constant K in several standard GANs, not only attaining significant performance gains, but also finding connections between K and training dynamics, particularly in low-gradient loss plateaus, with the common Adam optimizer.