Learning Compact Neural Networks with Regularization
This work addresses the need for cost-efficient and generalizable neural networks, though it appears incremental as it builds on existing regularization techniques.
The paper tackles the problem of learning compact neural networks efficiently by proposing regularized gradient descent algorithms, showing that near-optimal sample complexity is sufficient for local linear convergence once data exceeds the covering dimension.
Proper regularization is critical for speeding up training, improving generalization performance, and learning compact models that are cost efficient. We propose and analyze regularized gradient descent algorithms for learning shallow neural networks. Our framework is general and covers weight-sharing (convolutional networks), sparsity (network pruning), and low-rank constraints among others. We first introduce covering dimension to quantify the complexity of the constraint set and provide insights on the generalization properties. Then, we show that proposed algorithms become well-behaved and local linear convergence occurs once the amount of data exceeds the covering dimension. Overall, our results demonstrate that near-optimal sample complexity is sufficient for efficient learning and illustrate how regularization can be beneficial to learn over-parameterized networks.