On the Universality of Volume-Preserving and Coupling-Based Normalizing Flows
This work addresses a theoretical gap for researchers in machine learning, particularly in generative modeling, by clarifying the limitations and capabilities of different flow architectures, though it is incremental in building on existing empirical wisdom.
The authors tackled the problem of understanding the expressive power of normalizing flows, showing that well-conditioned coupling-based flows like RealNVP are universal, while volume-preserving flows are not, and they provided a fix for the latter.
We present a novel theoretical framework for understanding the expressive power of normalizing flows. Despite their prevalence in scientific applications, a comprehensive understanding of flows remains elusive due to their restricted architectures. Existing theorems fall short as they require the use of arbitrarily ill-conditioned neural networks, limiting practical applicability. We propose a distributional universality theorem for well-conditioned coupling-based normalizing flows such as RealNVP. In addition, we show that volume-preserving normalizing flows are not universal, what distribution they learn instead, and how to fix their expressivity. Our results support the general wisdom that affine and related couplings are expressive and in general outperform volume-preserving flows, bridging a gap between empirical results and theoretical understanding.