The layer-wise L1 Loss Landscape of Neural Nets is more complex around local minima
This incremental work could inform the design of gradient-descent algorithms to improve convergence rates for neural network training.
The authors investigated the L1 loss landscape of ReLU neural networks, finding that near local minima, loss decays exponentially slowly and vertex density increases sharply, enabling estimation of the minimum's loss level before reaching it.
For fixed training data and network parameters in the other layers the L1 loss of a ReLU neural network as a function of the first layer's parameters is a piece-wise affine function. We use the Deep ReLU Simplex algorithm to iteratively minimize the loss monotonically on adjacent vertices and analyze the trajectory of these vertex positions. We empirically observe that in a neighbourhood around a local minimum, the iterations behave differently such that conclusions on loss level and proximity of the local minimum can be made before it has been found: Firstly the loss seems to decay exponentially slow at iterated adjacent vertices such that the loss level at the local minimum can be estimated from the loss levels of subsequently iterated vertices, and secondly we observe a strong increase of the vertex density around local minima. This could have far-reaching consequences for the design of new gradient-descent algorithms that might improve convergence rate by exploiting these facts.