$(f,Γ)$-Divergences: Interpolating between $f$-Divergences and Integral Probability Metrics
This work addresses the problem of comparing complex probability distributions in machine learning, offering a more flexible divergence for researchers and practitioners, though it is incremental as it builds on existing divergence concepts.
The paper introduces $(f,\\Gamma)$-divergences, a framework that unifies $f$-divergences and integral probability metrics to compare probability distributions, enabling applications like improved training of generative adversarial networks for heavy-tailed or non-absolutely continuous data, with demonstrated advantages in image generation over gradient-penalized Wasserstein GAN.
We develop a rigorous and general framework for constructing information-theoretic divergences that subsume both $f$-divergences and integral probability metrics (IPMs), such as the $1$-Wasserstein distance. We prove under which assumptions these divergences, hereafter referred to as $(f,Γ)$-divergences, provide a notion of `distance' between probability measures and show that they can be expressed as a two-stage mass-redistribution/mass-transport process. The $(f,Γ)$-divergences inherit features from IPMs, such as the ability to compare distributions which are not absolutely continuous, as well as from $f$-divergences, namely the strict concavity of their variational representations and the ability to control heavy-tailed distributions for particular choices of $f$. When combined, these features establish a divergence with improved properties for estimation, statistical learning, and uncertainty quantification applications. Using statistical learning as an example, we demonstrate their advantage in training generative adversarial networks (GANs) for heavy-tailed, not-absolutely continuous sample distributions. We also show improved performance and stability over gradient-penalized Wasserstein GAN in image generation.