Neural PCA for Flow-Based Representation Learning
This addresses the need for better unsupervised representation learning in generative models, though it is incremental as it builds on existing flow-based methods.
The paper tackles the problem of whether normalizing flows provide effective representations for downstream tasks by proposing Neural-PCA, which captures principal components in descending order without using labels, resulting in 5%-10% performance improvements in downstream tasks.
Of particular interest is to discover useful representations solely from observations in an unsupervised generative manner. However, the question of whether existing normalizing flows provide effective representations for downstream tasks remains mostly unanswered despite their strong ability for sample generation and density estimation. This paper investigates this problem for such a family of generative models that admits exact invertibility. We propose Neural Principal Component Analysis (Neural-PCA) that operates in full dimensionality while capturing principal components in \emph{descending} order. Without exploiting any label information, the principal components recovered store the most informative elements in their \emph{leading} dimensions and leave the negligible in the \emph{trailing} ones, allowing for clear performance improvements of $5\%$-$10\%$ in downstream tasks. Such improvements are empirically found consistent irrespective of the number of latent trailing dimensions dropped. Our work suggests that necessary inductive bias be introduced into generative modelling when representation quality is of interest.