Information Geometry for the Working Information Theorist
It serves as an introductory guide for information theorists unfamiliar with information geometry, highlighting its relevance to recent developments in the field.
The paper provides an overview of information geometry, explaining its core concepts like divergences and geodesics on statistical manifolds to introduce information theorists to this interdisciplinary field, which has applications in areas such as deep learning and signal processing.
Information geometry is a study of statistical manifolds, that is, spaces of probability distributions from a geometric perspective. Its classical information-theoretic applications relate to statistical concepts such as Fisher information, sufficient statistics, and efficient estimators. Today, information geometry has emerged as an interdisciplinary field that finds applications in diverse areas such as radar sensing, array signal processing, quantum physics, deep learning, and optimal transport. This article presents an overview of essential information geometry to initiate an information theorist, who may be unfamiliar with this exciting area of research. We explain the concepts of divergences on statistical manifolds, generalized notions of distances, orthogonality, and geodesics, thereby paving the way for concrete applications and novel theoretical investigations. We also highlight some recent information-geometric developments, which are of interest to the broader information theory community.