Information-theoretic Quantification of High-order Feature Effects in Classification Problems
This work addresses the challenge of interpreting complex feature interactions in machine learning models, particularly for domains like genomics, though it is incremental as it builds on existing methods.
The authors tackled the problem of quantifying high-order feature interactions in classification by extending the Hi-Fi method with an information-theoretic approach using Conditional Mutual Information, and results showed accurate recovery of theoretical patterns on synthetic and real-world gene expression data.
Understanding the contribution of individual features in predictive models remains a central goal in interpretable machine learning, and while many model-agnostic methods exist to estimate feature importance, they often fall short in capturing high-order interactions and disentangling overlapping contributions. In this work, we present an information-theoretic extension of the High-order interactions for Feature importance (Hi-Fi) method, leveraging Conditional Mutual Information (CMI) estimated via a k-Nearest Neighbor (kNN) approach working on mixed discrete and continuous random variables. Our framework decomposes feature contributions into unique, synergistic, and redundant components, offering a richer, model-independent understanding of their predictive roles. We validate the method using synthetic datasets with known Gaussian structures, where ground truth interaction patterns are analytically derived, and further test it on non-Gaussian and real-world gene expression data from TCGA-BRCA. Results indicate that the proposed estimator accurately recovers theoretical and expected findings, providing a potential use case for developing feature selection algorithms or model development based on interaction analysis.