Taming neural networks with TUSLA: Non-convex learning via adaptive stochastic gradient Langevin algorithms
This addresses the challenge of unstable optimization in neural networks for researchers and practitioners, though it is incremental as it builds on existing taming and SGLD methods.
The authors tackled the problem of training neural networks with non-convex loss functions and superlinear gradient growth by proposing TUSLA, a tamed variant of stochastic gradient Langevin dynamics, which provided finite-time guarantees for finding approximate minimizers of empirical and population risks, with numerical experiments confirming its superiority over vanilla SGLD.
Artificial neural networks (ANNs) are typically highly nonlinear systems which are finely tuned via the optimization of their associated, non-convex loss functions. In many cases, the gradient of any such loss function has superlinear growth, making the use of the widely-accepted (stochastic) gradient descent methods, which are based on Euler numerical schemes, problematic. We offer a new learning algorithm based on an appropriately constructed variant of the popular stochastic gradient Langevin dynamics (SGLD), which is called tamed unadjusted stochastic Langevin algorithm (TUSLA). We also provide a nonasymptotic analysis of the new algorithm's convergence properties in the context of non-convex learning problems with the use of ANNs. Thus, we provide finite-time guarantees for TUSLA to find approximate minimizers of both empirical and population risks. The roots of the TUSLA algorithm are based on the taming technology for diffusion processes with superlinear coefficients as developed in \citet{tamed-euler, SabanisAoAP} and for MCMC algorithms in \citet{tula}. Numerical experiments are presented which confirm the theoretical findings and illustrate the need for the use of the new algorithm in comparison to vanilla SGLD within the framework of ANNs.