Spurious samples in deep generative models: bug or feature?
This challenges traditional views in generative modeling by providing evidence that spurious samples may be inherent to deep generative nets, which could impact researchers and practitioners in machine learning.
The paper investigates whether spurious samples in deep generative models can be eliminated without reducing coverage of the generating distribution, and experimentally shows this is not possible without losing the ability to model some test samples, suggesting these samples are a feature rather than a bug.
Traditional wisdom in generative modeling literature is that spurious samples that a model can generate are errors and they should be avoided. Recent research, however, has shown interest in studying or even exploiting such samples instead of eliminating them. In this paper, we ask the question whether such samples can be eliminated all together without sacrificing coverage of the generating distribution. For the class of models we consider, we experimentally demonstrate that this is not possible without losing the ability to model some of the test samples. While our results need to be confirmed on a broader set of model families, these initial findings provide partial evidence that spurious samples share structural properties with the learned dataset, which, in turn, suggests they are not simply errors but a feature of deep generative nets.