Ordering Dimensions with Nested Dropout Normalizing Flows
This addresses a limitation in representation learning for normalizing flows, but appears incremental as it builds on existing manifold-constrained flows without defining off-manifold densities.
The paper tackles the problem of learning low-dimensional, semantically meaningful representations with normalizing flows, which typically require matching latent and output dimensions, by introducing flows with ordered latent variables that define a sequence of manifolds close to the data, and notes a trade-off between flow likelihood and ordering quality.
The latent space of normalizing flows must be of the same dimensionality as their output space. This constraint presents a problem if we want to learn low-dimensional, semantically meaningful representations. Recent work has provided compact representations by fitting flows constrained to manifolds, but hasn't defined a density off that manifold. In this work we consider flows with full support in data space, but with ordered latent variables. Like in PCA, the leading latent dimensions define a sequence of manifolds that lie close to the data. We note a trade-off between the flow likelihood and the quality of the ordering, depending on the parameterization of the flow.