Ensembled sparse-input hierarchical networks for high-dimensional datasets
This addresses the challenge of making neural networks practical for high-dimensional data analysis, though it appears incremental as it builds on existing methods with modifications.
The authors tackled the problem of neural networks overfitting and requiring extensive hyperparameter tuning in high-dimensional, small-sample settings by proposing EASIER-net, which uses L1 penalties to prune network structure and adaptively selects architectures, achieving higher prediction accuracy than off-the-shelf methods on real-world datasets.
Neural networks have seen limited use in prediction for high-dimensional data with small sample sizes, because they tend to overfit and require tuning many more hyperparameters than existing off-the-shelf machine learning methods. With small modifications to the network architecture and training procedure, we show that dense neural networks can be a practical data analysis tool in these settings. The proposed method, Ensemble by Averaging Sparse-Input Hierarchical networks (EASIER-net), appropriately prunes the network structure by tuning only two L1-penalty parameters, one that controls the input sparsity and another that controls the number of hidden layers and nodes. The method selects variables from the true support if the irrelevant covariates are only weakly correlated with the response; otherwise, it exhibits a grouping effect, where strongly correlated covariates are selected at similar rates. On a collection of real-world datasets with different sizes, EASIER-net selected network architectures in a data-adaptive manner and achieved higher prediction accuracy than off-the-shelf methods on average.