Understanding Likelihood of Normalizing Flow and Image Complexity through the Lens of Out-of-Distribution Detection
This addresses a critical reliability problem in safety-critical applications like OOD detection for machine learning practitioners, though it is incremental as it builds on prior observations of DGM limitations.
The paper investigates why deep generative models, specifically Normalizing Flows, assign higher likelihoods to out-of-distribution images than to training data, proposing that less complex images concentrate in high-density latent regions, and experimentally validates this across five architectures, showing the issue can be mitigated by accounting for image complexity.
Out-of-distribution (OOD) detection is crucial to safety-critical machine learning applications and has been extensively studied. While recent studies have predominantly focused on classifier-based methods, research on deep generative model (DGM)-based methods have lagged relatively. This disparity may be attributed to a perplexing phenomenon: DGMs often assign higher likelihoods to unknown OOD inputs than to their known training data. This paper focuses on explaining the underlying mechanism of this phenomenon. We propose a hypothesis that less complex images concentrate in high-density regions in the latent space, resulting in a higher likelihood assignment in the Normalizing Flow (NF). We experimentally demonstrate its validity for five NF architectures, concluding that their likelihood is untrustworthy. Additionally, we show that this problem can be alleviated by treating image complexity as an independent variable. Finally, we provide evidence of the potential applicability of our hypothesis in another DGM, PixelCNN++.