A Review of Change of Variable Formulas for Generative Modeling
This is an incremental review that organizes existing knowledge for researchers in generative modeling, aiding in understanding and application of these formulas.
The paper tackles the problem of scattered information on change-of-variable formulas in generative modeling by providing a systematic review that collects 28 formulas, reveals relationships between methods, clarifies distinctions, and identifies research gaps.
Change-of-variables (CoV) formulas allow to reduce complicated probability densities to simpler ones by a learned transformation with tractable Jacobian determinant. They are thus powerful tools for maximum-likelihood learning, Bayesian inference, outlier detection, model selection, etc. CoV formulas have been derived for a large variety of model types, but this information is scattered over many separate works. We present a systematic treatment from the unifying perspective of encoder/decoder architectures, which collects 28 CoV formulas in a single place, reveals interesting relationships between seemingly diverse methods, emphasizes important distinctions that are not always clear in the literature, and identifies surprising gaps for future research.