Encoding prior knowledge in the structure of the likelihood
This addresses inference challenges in hierarchical models for machine learning and statistics, offering a method to improve computational efficiency, though it appears incremental as it builds on existing reparametrization techniques.
The paper tackles the problem of strong dependencies in deep hierarchical models by transforming parameters to flatten hierarchies and encode prior knowledge into the likelihood, resulting in faster inference that adapts to data informativeness, with performance improvements demonstrated in Gaussian process regression.
The inference of deep hierarchical models is problematic due to strong dependencies between the hierarchies. We investigate a specific transformation of the model parameters based on the multivariate distributional transform. This transformation is a special form of the reparametrization trick, flattens the hierarchy and leads to a standard Gaussian prior on all resulting parameters. The transformation also transfers all the prior information into the structure of the likelihood, hereby decoupling the transformed parameters a priori from each other. A variational Gaussian approximation in this standardized space will be excellent in situations of relatively uninformative data. Additionally, the curvature of the log-posterior is well-conditioned in directions that are weakly constrained by the data, allowing for fast inference in such a scenario. In an example we perform the transformation explicitly for Gaussian process regression with a priori unknown correlation structure. Deep models are inferred rapidly in highly and slowly in poorly informed situations. The flat model show exactly the opposite performance pattern. A synthesis of both, the deep and the flat perspective, provides their combined advantages and overcomes the individual limitations, leading to a faster inference.