Benjamin Gess

Benjamin Gess, Benoît Perthame, Panagiotis E. Souganidis

We develop a semi-discretization approximation for scalar conservation laws with multiple rough time dependence in inhomogeneous fluxes. The method is based on Brenier's transport-collapse algorithm and uses characteristics defined in the setting of rough paths. We prove strong $L^1$-convergence for inhomogeneous fluxes and provide a rate of convergence for homogeneous one's. The approximation scheme as well as the proofs are based on the recently developed theory of pathwise entropy solutions and uses the kinetic formulation which allows to define globally the (rough) characteristics.

Benjamin Gess, Sebastian Kassing

We prove explicit bounds on the exponential rate of convergence for the momentum stochastic gradient descent scheme (MSGD) for arbitrary, fixed hyperparameters (learning rate, friction parameter) and its continuous-in-time counterpart in the context of non-convex optimization. In the small step-size regime and in the case of flat minima or large noise intensities, these bounds prove faster convergence of MSGD compared to plain stochastic gradient descent (SGD). The results are shown for objective functions satisfying a local Polyak-Lojasiewicz inequality and under assumptions on the variance of MSGD that are satisfied in overparametrized settings. Moreover, we analyze the optimal choice of the friction parameter and show that the MSGD process almost surely converges to a local minimum.

Benjamin Gess, Rishabh S. Gvalani, Vitalii Konarovskyi

The convergence of stochastic interacting particle systems in the mean-field limit to solutions of conservative stochastic partial differential equations is established, with optimal rate of convergence. As a second main result, a quantitative central limit theorem for such SPDEs is derived, again, with optimal rate of convergence. The results apply, in particular, to the convergence in the mean-field scaling of stochastic gradient descent dynamics in overparametrized, shallow neural networks to solutions of SPDEs. It is shown that the inclusion of fluctuations in the limiting SPDE improves the rate of convergence, and retains information about the fluctuations of stochastic gradient descent in the continuum limit.

Benjamin Gess, Sebastian Kassing, Vitalii Konarovskyi

We propose new limiting dynamics for stochastic gradient descent in the small learning rate regime called stochastic modified flows. These SDEs are driven by a cylindrical Brownian motion and improve the so-called stochastic modified equations by having regular diffusion coefficients and by matching the multi-point statistics. As a second contribution, we introduce distribution dependent stochastic modified flows which we prove to describe the fluctuating limiting dynamics of stochastic gradient descent in the small learning rate - infinite width scaling regime.

Benjamin Fehrman, Benjamin Gess, Arnulf Jentzen

We prove the local convergence to minima and estimates on the rate of convergence for the stochastic gradient descent method in the case of not necessarily globally convex nor contracting objective functions. In particular, the results are applicable to simple objective functions arising in machine learning.