On the Rényi Cross-Entropy
This work provides theoretical insights into a generalization of cross-entropy used in deep learning, but it is incremental as it focuses on analytical derivations without new applications.
The authors analyzed the properties of the Rényi cross-entropy measure, deriving closed-form expressions for cases involving exponential family distributions and cross-entropy rates for Gaussian processes and Markov sources.
The Rényi cross-entropy measure between two distributions, a generalization of the Shannon cross-entropy, was recently used as a loss function for the improved design of deep learning generative adversarial networks. In this work, we examine the properties of this measure and derive closed-form expressions for it when one of the distributions is fixed and when both distributions belong to the exponential family. We also analytically determine a formula for the cross-entropy rate for stationary Gaussian processes and for finite-alphabet Markov sources.