What needles do sparse neural networks find in nonlinear haystacks
This work addresses a computational bottleneck in sparse neural network training for scenarios with small or noisy datasets, such as in genomics, but it is incremental as it generalizes an existing linear method to nonlinear models.
The authors tackled the problem of selecting the penalty parameter for sparse neural networks without cross-validation, which is computationally expensive and unsuitable for small datasets. They established a theoretical foundation for this selection and demonstrated its effectiveness through Monte Carlo simulations.
Using a sparsity inducing penalty in artificial neural networks (ANNs) avoids over-fitting, especially in situations where noise is high and the training set is small in comparison to the number of features. For linear models, such an approach provably also recovers the important features with high probability in regimes for a well-chosen penalty parameter. The typical way of setting the penalty parameter is by splitting the data set and performing the cross-validation, which is (1) computationally expensive and (2) not desirable when the data set is already small to be further split (for example, whole-genome sequence data). In this study, we establish the theoretical foundation to select the penalty parameter without cross-validation based on bounding with a high probability the infinite norm of the gradient of the loss function at zero under the zero-feature assumption. Our approach is a generalization of the universal threshold of Donoho and Johnstone (1994) to nonlinear ANN learning. We perform a set of comprehensive Monte Carlo simulations on a simple model, and the numerical results show the effectiveness of the proposed approach.