The Promises and Pitfalls of Deep Kernel Learning
This addresses reliability issues in uncertainty estimation for practitioners using deep kernel learning, though it is incremental as it builds on existing methods.
The paper identifies that deep kernel learning models can overfit despite using marginal likelihood optimization, showing this overfitting can be worse than non-Bayesian methods in certain scenarios, and finds that a fully Bayesian approach rectifies these failures, leading to performance improvements over standard neural networks and Gaussian processes.
Deep kernel learning (DKL) and related techniques aim to combine the representational power of neural networks with the reliable uncertainty estimates of Gaussian processes. One crucial aspect of these models is an expectation that, because they are treated as Gaussian process models optimized using the marginal likelihood, they are protected from overfitting. However, we identify situations where this is not the case. We explore this behavior, explain its origins and consider how it applies to real datasets. Through careful experimentation on the UCI, CIFAR-10, and the UTKFace datasets, we find that the overfitting from overparameterized maximum marginal likelihood, in which the model is "somewhat Bayesian", can in certain scenarios be worse than that from not being Bayesian at all. We explain how and when DKL can still be successful by investigating optimization dynamics. We also find that failures of DKL can be rectified by a fully Bayesian treatment, which leads to the desired performance improvements over standard neural networks and Gaussian processes.