A phase transition for finding needles in nonlinear haystacks with LASSO artificial neural networks
This work addresses the challenge of feature selection in nonlinear, high-dimensional settings for machine learning practitioners, representing an incremental advance by adapting LASSO-like sparsity methods to neural networks.
The paper tackles the problem of identifying sparse nonlinear associations (needles) in high-dimensional data using artificial neural networks, and demonstrates a phase transition in the probability of retrieving these features with a new sparsity-inducing penalty, outperforming other ANN learners.
To fit sparse linear associations, a LASSO sparsity inducing penalty with a single hyperparameter provably allows to recover the important features (needles) with high probability in certain regimes even if the sample size is smaller than the dimension of the input vector (haystack). More recently learners known as artificial neural networks (ANN) have shown great successes in many machine learning tasks, in particular fitting nonlinear associations. Small learning rate, stochastic gradient descent algorithm and large training set help to cope with the explosion in the number of parameters present in deep neural networks. Yet few ANN learners have been developed and studied to find needles in nonlinear haystacks. Driven by a single hyperparameter, our ANN learner, like for sparse linear associations, exhibits a phase transition in the probability of retrieving the needles, which we do not observe with other ANN learners. To select our penalty parameter, we generalize the universal threshold of Donoho and Johnstone (1994) which is a better rule than the conservative (too many false detections) and expensive cross-validation. In the spirit of simulated annealing, we propose a warm-start sparsity inducing algorithm to solve the high-dimensional, non-convex and non-differentiable optimization problem. We perform precise Monte Carlo simulations to show the effectiveness of our approach.