On Signal-to-Noise Ratio Issues in Variational Inference for Deep Gaussian Processes
This addresses a specific training bottleneck for researchers and practitioners using DGPs, offering an incremental fix to improve reliability and performance.
The paper tackles the problem of signal-to-noise ratio (SNR) issues in gradient estimates for training Deep Gaussian Processes (DGPs) with importance-weighted variational inference, showing that these estimates degrade to noise with too many importance samples. It adapts doubly reparameterized gradient estimators to DGPs, completely remedying the SNR issue and leading to consistent improvements in predictive performance.
We show that the gradient estimates used in training Deep Gaussian Processes (DGPs) with importance-weighted variational inference are susceptible to signal-to-noise ratio (SNR) issues. Specifically, we show both theoretically and via an extensive empirical evaluation that the SNR of the gradient estimates for the latent variable's variational parameters decreases as the number of importance samples increases. As a result, these gradient estimates degrade to pure noise if the number of importance samples is too large. To address this pathology, we show how doubly reparameterized gradient estimators, originally proposed for training variational autoencoders, can be adapted to the DGP setting and that the resultant estimators completely remedy the SNR issue, thereby providing more reliable training. Finally, we demonstrate that our fix can lead to consistent improvements in the predictive performance of DGP models.