From Optimization Dynamics to Generalization Bounds via Łojasiewicz Gradient Inequality
This work addresses the fundamental challenge of linking optimization and generalization in machine learning, providing a framework applicable to various models, though it appears incremental as it builds on existing concepts like Łojasiewicz inequality.
The paper tackles the problem of connecting optimization dynamics to generalization bounds by analyzing gradient flow trajectories under a Uniform-LGI property, deriving convergence rates and generalization bounds for models like linear regression, kernel regression, and two-layer neural networks, with results that match or extend prior findings.
Optimization and generalization are two essential aspects of statistical machine learning. In this paper, we propose a framework to connect optimization with generalization by analyzing the generalization error based on the optimization trajectory under the gradient flow algorithm. The key ingredient of this framework is the Uniform-LGI, a property that is generally satisfied when training machine learning models. Leveraging the Uniform-LGI, we first derive convergence rates for gradient flow algorithm, then we give generalization bounds for a large class of machine learning models. We further apply our framework to three distinct machine learning models: linear regression, kernel regression, and two-layer neural networks. Through our approach, we obtain generalization estimates that match or extend previous results.