Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks
This work provides incremental insights into understanding the geometry of loss landscapes in deep learning, which could aid in optimization and training for researchers.
The paper investigates the loss landscape of deep neural networks by systematically categorizing and visualizing 1D and 2D geometric structures, such as v-basin and saddle surfaces, and proposes mining algorithms to uncover complex patterns like w-peak curves, with theoretical insights from the Hessian matrix to explain observed phenomena.
The loss landscape of deep neural networks (DNNs) is commonly considered complex and wildly fluctuated. However, an interesting observation is that the loss surfaces plotted along Gaussian noise directions are almost v-basin ones with the perturbed model lying on the basin. This motivates us to rethink whether the 1D or 2D subspace could cover more complex local geometry structures, and how to mine the corresponding perturbation directions. This paper systematically and gradually categorizes the 1D curves from simple to complex, including v-basin, v-side, w-basin, w-peak, and vvv-basin curves. Notably, the latter two types are already hard to obtain via the intuitive construction of specific perturbation directions, and we need to propose proper mining algorithms to plot the corresponding 1D curves. Combining these 1D directions, various types of 2D surfaces are visualized such as the saddle surfaces and the bottom of a bottle of wine that are only shown by demo functions in previous works. Finally, we propose theoretical insights from the lens of the Hessian matrix to explain the observed several interesting phenomena.