Identifying Generalization Properties in Neural Networks
This work addresses a foundational challenge in machine learning by providing theoretical insights into generalization, which is crucial for improving model reliability across various applications.
The paper tackles the problem of understanding generalization in neural networks by proving a connection between generalization ability and local properties like the Hessian, Lipschitz constant, and parameter scales, and proposes a metric and algorithm to optimize generalization based on this proof.
While it has not yet been proven, empirical evidence suggests that model generalization is related to local properties of the optima which can be described via the Hessian. We connect model generalization with the local property of a solution under the PAC-Bayes paradigm. In particular, we prove that model generalization ability is related to the Hessian, the higher-order "smoothness" terms characterized by the Lipschitz constant of the Hessian, and the scales of the parameters. Guided by the proof, we propose a metric to score the generalization capability of the model, as well as an algorithm that optimizes the perturbed model accordingly.