Comparing Model-agnostic Feature Selection Methods through Relative Efficiency
This work addresses feature selection for machine learning practitioners, but it is incremental as it compares existing methods rather than introducing new ones.
The paper tackled the challenge of comparing model-agnostic feature selection methods, specifically Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO), by providing a theoretical and empirical analysis. The results showed that GCM-related methods generally outperform LOCO under suitable conditions, with quantified asymptotic relative efficiency.
Feature selection and importance estimation in a model-agnostic setting is an ongoing challenge of significant interest. Wrapper methods are commonly used because they are typically model-agnostic, even though they are computationally intensive. In this paper, we focus on feature selection methods related to the Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO) estimation, and provide a comparison based on relative efficiency. In particular, we present a theoretical comparison under three model settings: linear models, non-linear additive models, and single index models that mimic a single-layer neural network. We complement this with extensive simulations and real data examples. Our theoretical results, along with empirical findings, demonstrate that GCM-related methods generally outperform LOCO under suitable regularity conditions. Furthermore, we quantify the asymptotic relative efficiency of these approaches. Our simulations and real data analysis include widely used machine learning methods such as neural networks and gradient boosting trees.