Nicolas Brosse

Nicolas Brosse, Arnak S. Dalalyan

We study the small-noise asymptotics of Euclidean heat regularizations of probability measures supported on manifolds with corners. Near a boundary or corner stratum, the relevant regime is a conical boundary layer in which the observation point approaches the stratum at the same scale as the Gaussian smoothing parameter. After rescaling this layer, the support is replaced to leading order by its inward tangent cone. We prove a two-term expansion for the heat-regularized density in this regime. The leading coefficient is the Gaussian mass of the linearized cone, weighted by the density on the support and by the adapted corner Jacobian; the first correction records the variation of the density, the Jacobian, and the quadratic geometry of the embedding. A localization argument then yields the corresponding expansion for the full heat regularization, with the nonlocal contribution exponentially small. From this density expansion we derive logarithmic asymptotics and uniform expansions for the score, the log-Hessian, and the scale derivative of the score. These formulas show how lower-dimensional support, boundary faces, corners, and curvature are encoded in the singular differential structure of small-noise Gaussian regularizations.

Nicolas Brosse, Carlos Riquelme, Alice Martin et al.

Uncertainty quantification for deep learning is a challenging open problem. Bayesian statistics offer a mathematically grounded framework to reason about uncertainties; however, approximate posteriors for modern neural networks still require prohibitive computational costs. We propose a family of algorithms which split the classification task into two stages: representation learning and uncertainty estimation. We compare four specific instances, where uncertainty estimation is performed via either an ensemble of Stochastic Gradient Descent or Stochastic Gradient Langevin Dynamics snapshots, an ensemble of bootstrapped logistic regressions, or via a number of Monte Carlo Dropout passes. We evaluate their performance in terms of \emph{selective} classification (risk-coverage), and their ability to detect out-of-distribution samples. Our experiments suggest there is limited value in adding multiple uncertainty layers to deep classifiers, and we observe that these simple methods strongly outperform a vanilla point-estimate SGD in some complex benchmarks like ImageNet.

Nicolas Brosse, Alain Durmus, Eric Moulines

Stochastic Gradient Langevin Dynamics (SGLD) has emerged as a key MCMC algorithm for Bayesian learning from large scale datasets. While SGLD with decreasing step sizes converges weakly to the posterior distribution, the algorithm is often used with a constant step size in practice and has demonstrated successes in machine learning tasks. The current practice is to set the step size inversely proportional to $N$ where $N$ is the number of training samples. As $N$ becomes large, we show that the SGLD algorithm has an invariant probability measure which significantly departs from the target posterior and behaves like Stochastic Gradient Descent (SGD). This difference is inherently due to the high variance of the stochastic gradients. Several strategies have been suggested to reduce this effect; among them, SGLD Fixed Point (SGLDFP) uses carefully designed control variates to reduce the variance of the stochastic gradients. We show that SGLDFP gives approximate samples from the posterior distribution, with an accuracy comparable to the Langevin Monte Carlo (LMC) algorithm for a computational cost sublinear in the number of data points. We provide a detailed analysis of the Wasserstein distances between LMC, SGLD, SGLDFP and SGD and explicit expressions of the means and covariance matrices of their invariant distributions. Our findings are supported by limited numerical experiments.