Local properties of neural networks through the lens of layer-wise Hessians
This work provides a foundational tool for diagnosing and designing deep neural networks by connecting optimization geometry with functional behavior, offering practical insights for improving architectures and training stability, though it is incremental in nature.
The paper tackles the problem of analyzing neural networks by introducing a methodology based on layer-wise Hessian matrices to characterize local geometry in the parameter space, showing that spectral properties of these Hessians reveal patterns related to overfitting, underparameterization, and expressivity, with results from 111 experiments across 37 datasets demonstrating consistent structural regularities and correlations with generalization performance.
We introduce a methodology for analyzing neural networks through the lens of layer-wise Hessian matrices. The local Hessian of each functional block (layer) is defined as the matrix of second derivatives of a scalar function with respect to the parameters of that layer. This concept provides a formal tool for characterizing the local geometry of the parameter space. We show that the spectral properties of local Hessians, such as the distribution of eigenvalues, reveal quantitative patterns associated with overfitting, underparameterization, and expressivity in neural network architectures. We conduct an extensive empirical study involving 111 experiments across 37 datasets. The results demonstrate consistent structural regularities in the evolution of local Hessians during training and highlight correlations between their spectra and generalization performance. These findings establish a foundation for using local geometric analysis to guide the diagnosis and design of deep neural networks. The proposed framework connects optimization geometry with functional behavior and offers practical insight for improving network architectures and training stability.