Why neural networks find simple solutions: the many regularizers of geometric complexity
This work addresses the challenge of model complexity control in deep learning for researchers and practitioners, offering a novel perspective but is incremental in unifying existing methods.
The paper tackles the problem of understanding complexity control in deep neural networks by introducing geometric complexity, a measure based on discrete Dirichlet energy, and demonstrates that various common training heuristics like parameter norm regularization and noise regularization all act to control this measure, providing a unifying framework.
In many contexts, simpler models are preferable to more complex models and the control of this model complexity is the goal for many methods in machine learning such as regularization, hyperparameter tuning and architecture design. In deep learning, it has been difficult to understand the underlying mechanisms of complexity control, since many traditional measures are not naturally suitable for deep neural networks. Here we develop the notion of geometric complexity, which is a measure of the variability of the model function, computed using a discrete Dirichlet energy. Using a combination of theoretical arguments and empirical results, we show that many common training heuristics such as parameter norm regularization, spectral norm regularization, flatness regularization, implicit gradient regularization, noise regularization and the choice of parameter initialization all act to control geometric complexity, providing a unifying framework in which to characterize the behavior of deep learning models.