The Lottery Ticket Hypothesis in Denoising: Towards Semantic-Driven Initialization
This addresses the issue of unreliable image generation for users of text-to-image models, though it is incremental as it builds on existing diffusion model frameworks.
The paper tackles the problem of generation failures in text-to-image diffusion models, where users often need to generate many images to get a satisfying result, by proposing that initial noise images contain 'winning tickets' that naturally denoise into specific content. The result shows that aggregating these winning tickets into initial noise effectively induces the model to generate specified objects at desired locations, with experiments verifying their properties and generalizability.
Text-to-image diffusion models allow users control over the content of generated images. Still, text-to-image generation occasionally leads to generation failure requiring users to generate dozens of images under the same text prompt before they obtain a satisfying result. We formulate the lottery ticket hypothesis in denoising: randomly initialized Gaussian noise images contain special pixel blocks (winning tickets) that naturally tend to be denoised into specific content independently. The generation failure in standard text-to-image synthesis is caused by the gap between optimal and actual spatial distribution of winning tickets in initial noisy images. To this end, we implement semantic-driven initial image construction creating initial noise from known winning tickets for each concept mentioned in the prompt. We conduct a series of experiments that verify the properties of winning tickets and demonstrate their generalizability across images and prompts. Our results show that aggregating winning tickets into the initial noise image effectively induce the model to generate the specified object at the corresponding location. Project Page: https://ut-mao.github.io/noise.github.io