Towards Quantifying the Hessian Structure of Neural Networks
This work addresses a theoretical gap in understanding Hessian structure for researchers, potentially informing optimization and analysis in large-scale models like LLMs, but it is incremental as it builds on prior empirical observations.
The paper tackled the unclear theoretical foundation of the near-block-diagonal Hessian structure in neural networks by revealing it stems from static architectural and dynamic training forces, and provided rigorous analysis showing that the number of classes C is a primary driver, with block-diagonal structure emerging as C becomes large.
Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal that the reported Hessian structure comes from a mixture of two forces: a ``static force'' rooted in the architecture design, and a ''dynamic force'' arisen from training. We then provide a rigorous theoretical analysis of ''static force'' at random initialization. We study linear models and 1-hidden-layer networks for classification tasks with $C$ classes. By leveraging random matrix theory, we compare the limit distributions of the diagonal and off-diagonal Hessian blocks and find that the block-diagonal structure arises as $C$ becomes large. Our findings reveal that $C$ is one primary driver of the near-block-diagonal structure. These results may shed new light on the Hessian structure of large language models (LLMs), which typically operate with a large $C$ exceeding $10^4$.