Regularization by Misclassification in ReLU Neural Networks
This addresses the problem of understanding implicit regularization in neural networks for researchers, but it is incremental as it builds on existing label noise and sparsity studies.
The paper investigates how training ReLU neural networks with label noise (a variant of SGD where labels are randomly changed) leads to sparser solutions, with fewer active neurons and reduced test error in some cases, as shown through experiments and theoretical analysis for the extreme case of p=1.
We study the implicit bias of ReLU neural networks trained by a variant of SGD where at each step, the label is changed with probability $p$ to a random label (label smoothing being a close variant of this procedure). Our experiments demonstrate that label noise propels the network to a sparse solution in the following sense: for a typical input, a small fraction of neurons are active, and the firing pattern of the hidden layers is sparser. In fact, for some instances, an appropriate amount of label noise does not only sparsify the network but further reduces the test error. We then turn to the theoretical analysis of such sparsification mechanisms, focusing on the extremal case of $p=1$. We show that in this case, the network withers as anticipated from experiments, but surprisingly, in different ways that depend on the learning rate and the presence of bias, with either weights vanishing or neurons ceasing to fire.