MINERVA: Mutual Information Neural Estimation for Supervised Feature Selection
This addresses feature selection for domains like fraud detection where target relationships depend on feature interactions, but it is incremental as it builds on neural estimation methods.
The paper tackles the problem of supervised feature selection by introducing MINERVA, a method that uses neural networks to estimate mutual information between features and targets, effectively capturing higher-order interactions. Experimental results on synthetic and fraud datasets show it performs exact solutions and captures complex dependencies.
Existing feature filters rely on statistical pair-wise dependence metrics to model feature-target relationships, but this approach may fail when the target depends on higher-order feature interactions rather than individual contributions. We introduce Mutual Information Neural Estimation Regularized Vetting Algorithm (MINERVA), a novel approach to supervised feature selection based on neural estimation of mutual information between features and targets. We paramaterize the approximation of mutual information with neural networks and perform feature selection using a carefully designed loss function augmented with sparsity-inducing regularizers. Our method is implemented in a two-stage process to decouple representation learning from feature selection, ensuring better generalization and a more accurate expression of feature importance. We present examples of ubiquitous dependency structures that are rarely captured in literature and show that our proposed method effectively captures these complex feature-target relationships by evaluating feature subsets as an ensemble. Experimental results on synthetic and real-life fraud datasets demonstrate the efficacy of our method and its ability to perform exact solutions.