The Inductive Bias of Restricted f-GANs
This provides theoretical insights into GANs for researchers in machine learning, though it is incremental as it builds on existing GAN frameworks.
The paper theoretically analyzes restricted f-GANs to understand how generative adversarial learning differs from classical statistical inference methods like maximum likelihood and method of moments, showing that for linear KL-GANs, the optimal generator yields a combination of both approaches rather than either one.
Generative adversarial networks are a novel method for statistical inference that have achieved much empirical success; however, the factors contributing to this success remain ill-understood. In this work, we attempt to analyze generative adversarial learning -- that is, statistical inference as the result of a game between a generator and a discriminator -- with the view of understanding how it differs from classical statistical inference solutions such as maximum likelihood inference and the method of moments. Specifically, we provide a theoretical characterization of the distribution inferred by a simple form of generative adversarial learning called restricted f-GANs -- where the discriminator is a function in a given function class, the distribution induced by the generator is restricted to lie in a pre-specified distribution class and the objective is similar to a variational form of the f-divergence. A consequence of our result is that for linear KL-GANs -- that is, when the discriminator is a linear function over some feature space and f corresponds to the KL-divergence -- the distribution induced by the optimal generator is neither the maximum likelihood nor the method of moments solution, but an interesting combination of both.