Robust model training and generalisation with Studentising flows
This work addresses robustness and generalisation issues in probabilistic modeling for machine learning practitioners, though it appears incremental as it builds on existing normalising flow methods.
The paper tackles the problem of improving normalising flows by incorporating robust statistical insights, specifically using fat-tailed latent distributions like multivariate Student's t instead of Gaussian, resulting in benefits such as reduced generalisation gap and improved held-out data likelihood, with experiments confirming efficacy.
Normalising flows are tractable probabilistic models that leverage the power of deep learning to describe a wide parametric family of distributions, all while remaining trainable using maximum likelihood. We discuss how these methods can be further improved based on insights from robust (in particular, resistant) statistics. Specifically, we propose to endow flow-based models with fat-tailed latent distributions such as multivariate Student's $t$, as a simple drop-in replacement for the Gaussian distribution used by conventional normalising flows. While robustness brings many advantages, this paper explores two of them: 1) We describe how using fatter-tailed base distributions can give benefits similar to gradient clipping, but without compromising the asymptotic consistency of the method. 2) We also discuss how robust ideas lead to models with reduced generalisation gap and improved held-out data likelihood. Experiments on several different datasets confirm the efficacy of the proposed approach in both regards.