Searching for Minimal Optimal Neural Networks
This work addresses the need for theoretical foundations in neural network pruning, which is important for researchers and practitioners in machine learning, though it is incremental as it builds on existing destructive techniques.
The paper tackles the problem of selecting appropriate neural network sizes to avoid overfitting by providing the first theoretical guarantee for the destructive approach using Adaptive group Lasso, proving it can reconstruct the correct number of hidden nodes in one-hidden-layer feedforward networks with high probability.
Large neural network models have high predictive power but may suffer from overfitting if the training set is not large enough. Therefore, it is desirable to select an appropriate size for neural networks. The destructive approach, which starts with a large architecture and then reduces the size using a Lasso-type penalty, has been used extensively for this task. Despite its popularity, there is no theoretical guarantee for this technique. Based on the notion of minimal neural networks, we posit a rigorous mathematical framework for studying the asymptotic theory of the destructive technique. We prove that Adaptive group Lasso is consistent and can reconstruct the correct number of hidden nodes of one-hidden-layer feedforward networks with high probability. To the best of our knowledge, this is the first theoretical result establishing for the destructive technique.