Shaken, and Stirred: Long-Range Dependencies Enable Robust Outlier Detection with PixelCNN++
This work addresses the critical need for robust outlier detection in real-world deep learning deployments, offering a lightweight solution that improves reliability on complex image data, though it is incremental in refining existing methods.
The paper tackled the problem of unreliable outlier detection using deep generative model likelihoods by proposing inexpensive bijective transformations to reduce low-level biases and leverage long-range dependencies in PixelCNN++. The result was state-of-the-art outlier detection performance on complex natural image datasets, with effectiveness also demonstrated on other generative models like flows and VAEs.
Reliable outlier detection is critical for real-world deployment of deep learning models. Although extensively studied, likelihoods produced by deep generative models have been largely dismissed as being impractical for outlier detection. First, deep generative model likelihoods are readily biased by low-level input statistics. Second, many recent solutions for correcting these biases are computationally expensive, or do not generalize well to complex, natural datasets. Here, we explore outlier detection with a state-of-the-art deep autoregressive model: PixelCNN++. We show that biases in PixelCNN++ likelihoods arise primarily from predictions based on local dependencies. We propose two families of bijective transformations -- ``stirring'' and ``shaking'' -- which ameliorate low-level biases and isolate the contribution of long-range dependencies to PixelCNN++ likelihoods. These transformations are inexpensive and readily computed at evaluation time. We test our approaches extensively with five grayscale and six natural image datasets and show that they achieve or exceed state-of-the-art outlier detection, particularly on datasets with complex, natural images. We also show that our solutions work well with other types of generative models (generative flows and variational autoencoders) and that their efficacy is governed by each model's reliance on local dependencies. In sum, lightweight remedies suffice to achieve robust outlier detection on image data with deep generative models.