Parametric Adversarial Divergences are Good Losses for Generative Modeling
This is an incremental position paper that addresses the problem of defining relevant statistical quantities for generative modeling tasks, particularly for researchers in machine learning.
The paper argues that parametric adversarial divergences, used in GANs, have distinct properties from nonparametric divergences like Jensen-Shannon, making them more suitable for learning high-dimensional distributions by being sensitive only to specific aspects of the distribution based on discriminator architecture, while nonparametric divergences may not correlate with sample quality.
Parametric adversarial divergences, which are a generalization of the losses used to train generative adversarial networks (GANs), have often been described as being approximations of their nonparametric counterparts, such as the Jensen-Shannon divergence, which can be derived under the so-called optimal discriminator assumption. In this position paper, we argue that despite being "non-optimal", parametric divergences have distinct properties from their nonparametric counterparts which can make them more suitable for learning high-dimensional distributions. A key property is that parametric divergences are only sensitive to certain aspects/moments of the distribution, which depend on the architecture of the discriminator and the loss it was trained with. In contrast, nonparametric divergences such as the Kullback-Leibler divergence are sensitive to moments ignored by the discriminator, but they do not necessarily correlate with sample quality (Theis et al., 2016). Similarly, we show that mutual information can lead to unintuitive interpretations, and explore more intuitive alternatives based on parametric divergences. We conclude that parametric divergences are a flexible framework for defining statistical quantities relevant to a specific modeling task.