Spread Divergence
This addresses a foundational issue in machine learning for researchers and practitioners working with complex distributions, though it appears incremental as it builds on existing divergence concepts.
The paper tackles the problem of divergences being undefined for distributions with different supports or undefined densities by defining a Spread Divergence that ensures existence under certain conditions, and demonstrates its application in training implicit generative models like ICA and deep networks.
For distributions $\mathbb{P}$ and $\mathbb{Q}$ with different supports or undefined densities, the divergence $\textrm{D}(\mathbb{P}||\mathbb{Q})$ may not exist. We define a Spread Divergence $\tilde{\textrm{D}}(\mathbb{P}||\mathbb{Q})$ on modified $\mathbb{P}$ and $\mathbb{Q}$ and describe sufficient conditions for the existence of such a divergence. We demonstrate how to maximize the discriminatory power of a given divergence by parameterizing and learning the spread. We also give examples of using a Spread Divergence to train implicit generative models, including linear models (Independent Components Analysis) and non-linear models (Deep Generative Networks).