Efficient Bayesian Updates for Deep Learning via Laplace Approximations
This addresses the challenge of efficiently updating deep learning models for applications with time or computational constraints, though it is incremental as it builds on existing Laplace approximation techniques.
The paper tackles the problem of costly retraining for deep neural networks when new data arrives by proposing a Bayesian update method using a last-layer Laplace approximation, achieving fast and competitive updates as an alternative to full retraining in evaluations across different data modalities.
Since training deep neural networks takes significant computational resources, extending the training dataset with new data is difficult, as it typically requires complete retraining. Moreover, specific applications do not allow costly retraining due to time or computational constraints. We address this issue by proposing a novel Bayesian update method for deep neural networks by using a last-layer Laplace approximation. Concretely, we leverage second-order optimization techniques on the Gaussian posterior distribution of a Laplace approximation, computing the inverse Hessian matrix in closed form. This way, our method allows for fast and effective updates upon the arrival of new data in a stationary setting. A large-scale evaluation study across different data modalities confirms that our updates are a fast and competitive alternative to costly retraining. Furthermore, we demonstrate its applicability in a deep active learning scenario by using our update to improve existing selection strategies.