Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source)
This is an incremental technical resource for practitioners and researchers aiming to analyze Hessian structure in LLMs.
The authors tackled the problem of computing the Hessian matrix for Large Language Models, which is infeasible in full due to size, by providing a guide and code to compute portions of it using PyTorch autograd, including the full diagonal via vector-Hessian products.
Computing the full Hessian matrix -- the matrix of second-order derivatives for an entire Large Language Model (LLM) is infeasible due to its sheer size. In this technical report, we aim to provide a comprehensive guide on how to accurately compute at least a small portion of the Hessian for LLMs using PyTorch autograd library. We also demonstrate how to compute the full diagonal of the Hessian matrix using multiple samples of vector-Hessian Products (HVPs). We hope that both this guide and the accompanying GitHub code will be valuable resources for practitioners and researchers interested in better understanding the behavior and structure of the Hessian in LLMs.