The Efficacy of $L_1$ Regularization in Two-Layer Neural Networks
This addresses the bias-variance tradeoff for neural network practitioners, offering theoretical insights into regularization as an alternative to neuron selection, though it appears incremental in its analytical approach.
The paper tackles the problem of selecting hidden neurons and obtaining tight statistical risk bounds in two-layer neural networks by showing that L1 regularization can control generalization error and sparsify input dimensions, achieving near minimax optimal risk and dimension-independent bounds.
A crucial problem in neural networks is to select the most appropriate number of hidden neurons and obtain tight statistical risk bounds. In this work, we present a new perspective towards the bias-variance tradeoff in neural networks. As an alternative to selecting the number of neurons, we theoretically show that $L_1$ regularization can control the generalization error and sparsify the input dimension. In particular, with an appropriate $L_1$ regularization on the output layer, the network can produce a statistical risk that is near minimax optimal. Moreover, an appropriate $L_1$ regularization on the input layer leads to a risk bound that does not involve the input data dimension. Our analysis is based on a new amalgamation of dimension-based and norm-based complexity analysis to bound the generalization error. A consequent observation from our results is that an excessively large number of neurons do not necessarily inflate generalization errors under a suitable regularization.