Online Laplace Model Selection Revisited
This work addresses a theoretical gap in Bayesian deep learning for model selection, though it is incremental as it revisits and corrects existing methods.
The paper tackled the issue of online Laplace methods violating key assumptions of the Laplace approximation in neural networks, showing that these methods target a variational bound on a mode-corrected evidence and share stationary points where parameters are maximum a posteriori and hyperparameters maximize evidence, with optimised hyperparameters preventing overfitting and outperforming validation-based early stopping on UCI regression datasets.
The Laplace approximation provides a closed-form model selection objective for neural networks (NN). Online variants, which optimise NN parameters jointly with hyperparameters, like weight decay strength, have seen renewed interest in the Bayesian deep learning community. However, these methods violate Laplace's method's critical assumption that the approximation is performed around a mode of the loss, calling into question their soundness. This work re-derives online Laplace methods, showing them to target a variational bound on a mode-corrected variant of the Laplace evidence which does not make stationarity assumptions. Online Laplace and its mode-corrected counterpart share stationary points where 1. the NN parameters are a maximum a posteriori, satisfying the Laplace method's assumption, and 2. the hyperparameters maximise the Laplace evidence, motivating online methods. We demonstrate that these optima are roughly attained in practise by online algorithms using full-batch gradient descent on UCI regression datasets. The optimised hyperparameters prevent overfitting and outperform validation-based early stopping.