Relative Entropy, Probabilistic Inference and AI
This is an incremental review paper that synthesizes existing knowledge on relative entropy for researchers in information theory and AI.
The paper reviews the properties of relative entropy and its role in probabilistic inference, discussing its potential applications in artificial intelligence.
Various properties of relative entropy have led to its widespread use in information theory. These properties suggest that relative entropy has a role to play in systems that attempt to perform inference in terms of probability distributions. In this paper, I will review some basic properties of relative entropy as well as its role in probabilistic inference. I will also mention briefly a few existing and potential applications of relative entropy to so-called artificial intelligence (AI).