Exploring Image Generation via Mutually Exclusive Probability Spaces and Local Correlation Hypothesis
This work addresses a fundamental limitation in image generation for AI researchers, though it appears incremental as it builds on existing probabilistic and autoregressive models.
The paper tackles the problem of generative models memorizing data instead of generating novel images by proposing the Mutually Exclusive Probability Space (MEPS) and Local Dependence Hypothesis (LDH) frameworks, and shows that increasing observation range in autoregressive models leads to memorization, with a Binary Latent Autoencoder (BL-AE) and $\gamma$-ARVM model demonstrating this effect.
A common assumption in probabilistic generative models for image generation is that learning the global data distribution suffices to generate novel images via sampling. We investigate the limitation of this core assumption, namely that learning global distributions leads to memorization rather than generative behavior. We propose two theoretical frameworks, the Mutually Exclusive Probability Space (MEPS) and the Local Dependence Hypothesis (LDH), for investigation. MEPS arises from the observation that deterministic mappings (e.g. neural networks) involving random variables tend to reduce overlap coefficients among involved random variables, thereby inducing exclusivity. We further propose a lower bound in terms of the overlap coefficient, and introduce a Binary Latent Autoencoder (BL-AE) that encodes images into signed binary latent representations. LDH formalizes dependence within a finite observation radius, which motivates our $γ$-Autoregressive Random Variable Model ($γ$-ARVM). $γ$-ARVM is an autoregressive model, with a variable observation range $γ$, that predicts a histogram for the next token. Using $γ$-ARVM, we observe that as the observation range increases, autoregressive models progressively shift toward memorization. In the limit of global dependence, the model behaves as a pure memorizer when operating on the binary latents produced by our BL-AE. Comprehensive experiments and discussions support our investigation.