Factorized Diffusion: Perceptual Illusions by Noise Decomposition
This provides a method for creating perceptual illusions in image generation, which is incremental as it builds on existing diffusion and compositional techniques.
The authors tackled the problem of controlling individual linear components of an image in diffusion models, enabling zero-shot generation of hybrid images that change appearance based on viewing conditions like distance, grayscale, or motion blur, with demonstrations including three-frequency subband decompositions.
Given a factorization of an image into a sum of linear components, we present a zero-shot method to control each individual component through diffusion model sampling. For example, we can decompose an image into low and high spatial frequencies and condition these components on different text prompts. This produces hybrid images, which change appearance depending on viewing distance. By decomposing an image into three frequency subbands, we can generate hybrid images with three prompts. We also use a decomposition into grayscale and color components to produce images whose appearance changes when they are viewed in grayscale, a phenomena that naturally occurs under dim lighting. And we explore a decomposition by a motion blur kernel, which produces images that change appearance under motion blurring. Our method works by denoising with a composite noise estimate, built from the components of noise estimates conditioned on different prompts. We also show that for certain decompositions, our method recovers prior approaches to compositional generation and spatial control. Finally, we show that we can extend our approach to generate hybrid images from real images. We do this by holding one component fixed and generating the remaining components, effectively solving an inverse problem.