The Design of Global Correlation Quantifiers and Continuous Notions of Statistical Sufficiency
This work provides conceptual clarity for researchers in information theory and statistics by ruling out alternative correlation quantifiers and offering a foundation for analyzing correlation preservation in transformations.
The paper tackles the problem of ranking joint probability distributions by their correlations, deriving a general functional called n-partite information that uniquely determines how inferential transformations affect correlations and enables quantification of non-binary statistical sufficiency.
Using first principles from inference, we design a set of functionals for the purposes of \textit{ranking} joint probability distributions with respect to their correlations. Starting with a general functional, we impose its desired behaviour through the \textit{Principle of Constant Correlations} (PCC), which constrains the correlation functional to behave in a consistent way under statistically independent inferential transformations. The PCC guides us in choosing the appropriate design criteria for constructing the desired functionals. Since the derivations depend on a choice of partitioning the variable space into $n$ disjoint subspaces, the general functional we design is the $n$-partite information (NPI), of which the \textit{total correlation} and \textit{mutual information} are special cases. Thus, these functionals are found to be uniquely capable of determining whether a certain class of inferential transformations, $ρ\xrightarrow{*}ρ'$, preserve, destroy or create correlations. This provides conceptual clarity by ruling out other possible global correlation quantifiers. Finally, the derivation and results allow us to quantify non-binary notions of statistical sufficency. Our results express what percentage of the correlations are preserved under a given inferential transformation or variable mapping.