GAIT: A Geometric Approach to Information Theory
This provides a versatile tool for domains like generative modeling and image processing, though it is an incremental extension of existing entropy concepts.
The paper tackles the problem of incorporating geometric relationships between symbols into information theory by introducing a geometry-aware entropy and divergence, achieving performance comparable to state-of-the-art Wasserstein distance methods with a closed-form expression for efficient computation.
We advocate the use of a notion of entropy that reflects the relative abundances of the symbols in an alphabet, as well as the similarities between them. This concept was originally introduced in theoretical ecology to study the diversity of ecosystems. Based on this notion of entropy, we introduce geometry-aware counterparts for several concepts and theorems in information theory. Notably, our proposed divergence exhibits performance on par with state-of-the-art methods based on the Wasserstein distance, but enjoys a closed-form expression that can be computed efficiently. We demonstrate the versatility of our method via experiments on a broad range of domains: training generative models, computing image barycenters, approximating empirical measures and counting modes.