Max-Sliced Mutual Information
This work addresses the need for scalable dependence measures in statistical learning, offering a method that bridges the gap between linear and high-order dependence for applications such as multi-view learning and fairness, though it is incremental in building on existing information-theoretic concepts.
The paper tackles the problem of quantifying dependence between high-dimensional random variables by proposing max-sliced mutual information (mSMI), a scalable generalization of canonical correlation analysis that captures intricate dependencies while being computationally efficient, and demonstrates its utility in tasks like independence testing and representation learning with consistent performance improvements over competing methods.
Quantifying the dependence between high-dimensional random variables is central to statistical learning and inference. Two classical methods are canonical correlation analysis (CCA), which identifies maximally correlated projected versions of the original variables, and Shannon's mutual information, which is a universal dependence measure that also captures high-order dependencies. However, CCA only accounts for linear dependence, which may be insufficient for certain applications, while mutual information is often infeasible to compute/estimate in high dimensions. This work proposes a middle ground in the form of a scalable information-theoretic generalization of CCA, termed max-sliced mutual information (mSMI). mSMI equals the maximal mutual information between low-dimensional projections of the high-dimensional variables, which reduces back to CCA in the Gaussian case. It enjoys the best of both worlds: capturing intricate dependencies in the data while being amenable to fast computation and scalable estimation from samples. We show that mSMI retains favorable structural properties of Shannon's mutual information, like variational forms and identification of independence. We then study statistical estimation of mSMI, propose an efficiently computable neural estimator, and couple it with formal non-asymptotic error bounds. We present experiments that demonstrate the utility of mSMI for several tasks, encompassing independence testing, multi-view representation learning, algorithmic fairness, and generative modeling. We observe that mSMI consistently outperforms competing methods with little-to-no computational overhead.