Improving the Linearized Laplace Approximation via Quadratic Approximations
This work addresses uncertainty quantification for deep learning practitioners, but it is incremental as it builds on existing methods with specific computational enhancements.
The paper tackles the problem of overconfident out-of-distribution predictions in deep neural networks by proposing the Quadratic Laplace Approximation (QLA) to improve uncertainty estimation over the Linearized Laplace Approximation (LLA), achieving modest yet consistent improvements on five regression datasets.
Deep neural networks (DNNs) often produce overconfident out-of-distribution predictions, motivating Bayesian uncertainty quantification. The Linearized Laplace Approximation (LLA) achieves this by linearizing the DNN and applying Laplace inference to the resulting model. Importantly, the linear model is also used for prediction. We argue this linearization in the posterior may degrade fidelity to the true Laplace approximation. To alleviate this problem, without increasing significantly the computational cost, we propose the Quadratic Laplace Approximation (QLA). QLA approximates each second order factor in the approximate Laplace log-posterior using a rank-one factor obtained via efficient power iterations. QLA is expected to yield a posterior precision closer to that of the full Laplace without forming the full Hessian, which is typically intractable. For prediction, QLA also uses the linearized model. Empirically, QLA yields modest yet consistent uncertainty estimation improvements over LLA on five regression datasets.