Efficient Deep Gaussian Process Models for Variable-Sized Input
This work addresses a bottleneck in Bayesian deep learning for researchers and practitioners dealing with non-standard data structures, offering a tractable solution for variable-sized inputs, though it is incremental as it builds on existing DRF methods.
The authors tackled the problem of scaling deep Gaussian processes (DGPs) for variable-sized inputs like trees and graphs, introducing GP-DRF, a model that combines GPs with deep random features to handle such data efficiently. Their experiments show GP-DRF outperforms standard GP and DRF models across many datasets and improves uncertainty quantification, with specific gains noted in Bhattacharyya distance assessments.
Deep Gaussian processes (DGP) have appealing Bayesian properties, can handle variable-sized data, and learn deep features. Their limitation is that they do not scale well with the size of the data. Existing approaches address this using a deep random feature (DRF) expansion model, which makes inference tractable by approximating DGPs. However, DRF is not suitable for variable-sized input data such as trees, graphs, and sequences. We introduce the GP-DRF, a novel Bayesian model with an input layer of GPs, followed by DRF layers. The key advantage is that the combination of GP and DRF leads to a tractable model that can both handle a variable-sized input as well as learn deep long-range dependency structures of the data. We provide a novel efficient method to simultaneously infer the posterior of GP's latent vectors and infer the posterior of DRF's internal weights and random frequencies. Our experiments show that GP-DRF outperforms the standard GP model and DRF model across many datasets. Furthermore, they demonstrate that GP-DRF enables improved uncertainty quantification compared to GP and DRF alone, with respect to a Bhattacharyya distance assessment. Source code is available at https://github.com/IssamLaradji/GP_DRF.