A Nonparametric Statistics Approach to Feature Selection in Deep Neural Networks with Theoretical Guarantees
This work solves the problem of identifying relevant features in complex, nonlinear models for researchers and practitioners in machine learning, offering theoretical guarantees but being incremental as it builds on existing nonparametric and neural network methods.
The paper addresses feature selection in deep neural networks for nonlinear functions with unknown relevant features, achieving selection consistency with sample size requirements of n = Ω(p^2) for low dimensions and n = Ω(s log p) for high-dimensional sparse scenarios, and demonstrates strong performance in simulations and real data.
This paper tackles the problem of feature selection in a highly challenging setting: $\mathbb{E}(y | \boldsymbol{x}) = G(\boldsymbol{x}_{\mathcal{S}_0})$, where $\mathcal{S}_0$ is the set of relevant features and $G$ is an unknown, potentially nonlinear function subject to mild smoothness conditions. Our approach begins with feature selection in deep neural networks, then generalizes the results to H{ö}lder smooth functions by exploiting the strong approximation capabilities of neural networks. Unlike conventional optimization-based deep learning methods, we reformulate neural networks as index models and estimate $\mathcal{S}_0$ using the second-order Stein's formula. This gradient-descent-free strategy guarantees feature selection consistency with a sample size requirement of $n = Ω(p^2)$, where $p$ is the feature dimension. To handle high-dimensional scenarios, we further introduce a screening-and-selection mechanism that achieves nonlinear selection consistency when $n = Ω(s \log p)$, with $s$ representing the sparsity level. Additionally, we refit a neural network on the selected features for prediction and establish performance guarantees under a relaxed sparsity assumption. Extensive simulations and real-data analyses demonstrate the strong performance of our method even in the presence of complex feature interactions.