Poincaré Inequality for Local Log-Polyak-Lojasiewicz Measures : Non-asymptotic Analysis in Low-temperature Regime
This provides theoretical insights into stochastic optimization for non-convex problems in machine learning, though it is incremental as it builds on existing Polyak-Łojasiewicz and Poincaré inequality frameworks.
The paper tackles the problem of understanding convergence in non-convex landscapes with connected minima, such as in over-parameterized deep learning, by analyzing Poincaré inequalities for log-Polyak-Łojasiewicz measures and showing that Langevin dynamics converges at a rate of Õ(1/ε) in the low-temperature regime.
Potential functions in highly pertinent applications, such as deep learning in over-parameterized regime, are empirically observed to admit non-isolated minima. To understand the convergence behavior of stochastic dynamics in such landscapes, we propose to study the class of \logPLmeasure\ measures $μ_ε\propto \exp(-V/ε)$, where the potential $V$ satisfies a local Polyak-Łojasiewicz (PŁ) inequality, and its set of local minima is provably \emph{connected}. Notably, potentials in this class can exhibit local maxima and we characterize its optimal set S to be a compact $\mathcal{C}^2$ \emph{embedding submanifold} of $\mathbb{R}^d$ without boundary. The \emph{non-contractibility} of S distinguishes our function class from the classical convex setting topologically. Moreover, the embedding structure induces a naturally defined Laplacian-Beltrami operator on S, and we show that its first non-trivial eigenvalue provides an \emph{$ε$-independent} lower bound for the \Poincare\ constant in the \Poincare\ inequality of $μ_ε$. As a direct consequence, Langevin dynamics with such non-convex potential $V$ and diffusion coefficient $ε$ converges to its equilibrium $μ_ε$ at a rate of $\tilde{\mathcal{O}}(1/ε)$, provided $ε$ is sufficiently small. Here $\tilde{\mathcal{O}}$ hides logarithmic terms.