Mitigating Out-of-Distribution Data Density Overestimation in Energy-Based Models
This addresses a specific issue in generative modeling for researchers, offering an incremental improvement over existing methods.
The paper tackled the problem of deep energy-based models (EBMs) overestimating density for out-of-distribution (OOD) data when trained with maximum likelihood estimation and short-run Langevin Monte Carlo, and proposed a Uniform Support Partitioning (USP) scheme that improved OOD detection performance on Fashion-MNIST.
Deep energy-based models (EBMs), which use deep neural networks (DNNs) as energy functions, are receiving increasing attention due to their ability to learn complex distributions. To train deep EBMs, the maximum likelihood estimation (MLE) with short-run Langevin Monte Carlo (LMC) is often used. While the MLE with short-run LMC is computationally efficient compared to an MLE with full Markov Chain Monte Carlo (MCMC), it often assigns high density to out-of-distribution (OOD) data. To address this issue, here we systematically investigate why the MLE with short-run LMC can converge to EBMs with wrong density estimates, and reveal that the heuristic modifications to LMC introduced by previous works were the main problem. We then propose a Uniform Support Partitioning (USP) scheme that optimizes a set of points to evenly partition the support of the EBM and then uses the resulting points to approximate the EBM-MLE loss gradient. We empirically demonstrate that USP avoids the pitfalls of short-run LMC, leading to significantly improved OOD data detection performance on Fashion-MNIST.