Hybrid Models with Deep and Invertible Features
This work addresses the problem of integrating generative and discriminative capabilities in probabilistic deep learning for researchers and practitioners, though it is incremental as it builds on existing invertible flow methods.
The authors tackled the challenge of combining exact density estimation with predictive accuracy in deep learning by proposing a hybrid model using a deep invertible transformation and a linear model, achieving similar accuracy to purely predictive models while enabling exact computation of joint densities.
We propose a neural hybrid model consisting of a linear model defined on a set of features computed by a deep, invertible transformation (i.e. a normalizing flow). An attractive property of our model is that both p(features), the density of the features, and p(targets | features), the predictive distribution, can be computed exactly in a single feed-forward pass. We show that our hybrid model, despite the invertibility constraints, achieves similar accuracy to purely predictive models. Moreover the generative component remains a good model of the input features despite the hybrid optimization objective. This offers additional capabilities such as detection of out-of-distribution inputs and enabling semi-supervised learning. The availability of the exact joint density p(targets, features) also allows us to compute many quantities readily, making our hybrid model a useful building block for downstream applications of probabilistic deep learning.