Inherent Noise in Gradient Based Methods
This addresses the problem of understanding generalization in neural networks for researchers, but it is incremental as it builds on prior work linking noise and robustness.
The paper investigates how gradient descent (GD) and stochastic gradient descent (SGD) introduce inherent noise during optimization due to simultaneous parameter updates, which penalizes models sensitive to weight perturbations, with effects being more pronounced for larger models and current batches.
Previous work has examined the ability of larger capacity neural networks to generalize better than smaller ones, even without explicit regularizers, by analyzing gradient based algorithms such as GD and SGD. The presence of noise and its effect on robustness to parameter perturbations has been linked to generalization. We examine a property of GD and SGD, namely that instead of iterating through all scalar weights in the network and updating them one by one, GD (and SGD) updates all the parameters at the same time. As a result, each parameter $w^i$ calculates its partial derivative at the stale parameter $\mathbf{w_t}$, but then suffers loss $\hat{L}(\mathbf{w_{t+1}})$. We show that this causes noise to be introduced into the optimization. We find that this noise penalizes models that are sensitive to perturbations in the weights. We find that penalties are most pronounced for batches that are currently being used to update, and are higher for larger models.