How regularization affects the geometry of loss functions
This work addresses the theoretical understanding of optimization landscapes in deep learning, which is incremental but important for researchers in machine learning theory.
The paper investigates how different regularization techniques, such as weight decay, influence the geometric properties of loss functions in neural networks, specifically whether they make the loss function Morse, a fundamental smoothness property.
What neural networks learn depends fundamentally on the geometry of the underlying loss function. We study how different regularizers affect the geometry of this function. One of the most basic geometric properties of a smooth function is whether it is Morse or not. For nonlinear deep neural networks, the unregularized loss function $L$ is typically not Morse. We consider several different regularizers, including weight decay, and study for which regularizers the regularized function $L_ε$ becomes Morse.