Deep Q-Exponential Processes
This work addresses the issue of sub-optimal regularization for inhomogeneous data like images in deep probabilistic modeling, representing an incremental improvement over existing methods.
The paper tackles the problem of over-smoothing in deep Gaussian processes (DGPs) by generalizing Q-exponential processes (Q-EPs) to deep Q-EPs, achieving improved regularization and expressiveness, with numerical comparisons showing advantages over state-of-the-art deep probabilistic models.
Motivated by deep neural networks, the deep Gaussian process (DGP) generalizes the standard GP by stacking multiple layers of GPs. Despite the enhanced expressiveness, GP, as an $L_2$ regularization prior, tends to be over-smooth and sub-optimal for inhomogeneous subjects, such as images with edges. Recently, Q-exponential process (Q-EP) has been proposed as an $L_q$ relaxation to GP and demonstrated with more desirable regularization properties through a parameter $q>0$ with $q=2$ corresponding to GP. Sharing the similar tractability of posterior and predictive distributions with GP, Q-EP can also be stacked to improve its modeling flexibility. In this paper, we generalize Q-EP to deep Q-EP to enjoy both proper regularization and improved expressiveness. The generalization is realized by introducing shallow Q-EP as a latent variable model and then building a hierarchy of the shallow Q-EP layers. Sparse approximation by inducing points and scalable variational strategy are applied to facilitate the inference. We demonstrate the numerical advantages of the proposed deep Q-EP model by comparing with multiple state-of-the-art deep probabilistic models.