Implicit Rugosity Regularization via Data Augmentation
This provides theoretical insight into generalization for deep learning practitioners, though it is incremental as it builds on existing regularization concepts.
The paper investigates how data augmentation implicitly regularizes deep networks by penalizing a novel measure of rugosity based on the tangent Hessian, showing that this reduces overfitting in overparameterized regimes.
Deep (neural) networks have been applied productively in a wide range of supervised and unsupervised learning tasks. Unlike classical machine learning algorithms, deep networks typically operate in the \emph{overparameterized} regime, where the number of parameters is larger than the number of training data points. Consequently, understanding the generalization properties and the role of (explicit or implicit) regularization in these networks is of great importance. In this work, we explore how the oft-used heuristic of \emph{data augmentation} imposes an {\em implicit regularization} penalty of a novel measure of the \emph{rugosity} or "roughness" based on the tangent Hessian of the function fit to the training data.