Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions
This work addresses a bottleneck in multimodal analysis for researchers and practitioners by improving PID computation, though it appears incremental as it builds on existing PID frameworks with specific optimizations.
The paper tackles the computational inefficiency and inaccuracy of partial information decomposition (PID) for continuous high-dimensional data by proposing Gaussian PID (GPID) with a gradient-based algorithm and information-preserving encoders for non-Gaussian data, achieving more accurate and efficient PID estimates than baselines in synthetic and real-world multimodal benchmarks.
The study of multimodality has garnered significant interest in fields where the analysis of interactions among multiple information sources can enhance predictive modeling, data fusion, and interpretability. Partial information decomposition (PID) has emerged as a useful information-theoretic framework to quantify the degree to which individual modalities independently, redundantly, or synergistically convey information about a target variable. However, existing PID methods depend on optimizing over a joint distribution constrained by estimated pairwise probability distributions, which are costly and inaccurate for continuous and high-dimensional modalities. Our first key insight is that the problem can be solved efficiently when the pairwise distributions are multivariate Gaussians, and we refer to this problem as Gaussian PID (GPID). We propose a new gradient-based algorithm that substantially improves the computational efficiency of GPID based on an alternative formulation of the underlying optimization problem. To generalize the applicability to non-Gaussian data, we learn information-preserving encoders to transform random variables of arbitrary input distributions into pairwise Gaussian random variables. Along the way, we resolved an open problem regarding the optimality of joint Gaussian solutions for GPID. Empirical validation in diverse synthetic examples demonstrates that our proposed method provides more accurate and efficient PID estimates than existing baselines. We further evaluate a series of large-scale multimodal benchmarks to show its utility in real-world applications of quantifying PID in multimodal datasets and selecting high-performing models.