Varying Manifolds in Diffusion: From Time-varying Geometries to Visual Saliency
This work addresses the need for more interpretable and controllable image editing techniques in generative AI, though it is incremental as it builds on existing diffusion model frameworks.
The paper tackles the problem of understanding and controlling image manipulation in diffusion models by introducing the generation rate, which measures local manifold deformation over time and correlates with visual saliency. The results show that their method consistently leads to better manipulation outcomes, such as semantic transfer and object removal, compared to recent baselines.
Deep generative models learn the data distribution, which is concentrated on a low-dimensional manifold. The geometric analysis of distribution transformation provides a better understanding of data structure and enables a variety of applications. In this paper, we study the geometric properties of the diffusion model, whose forward diffusion process and reverse generation process construct a series of distributions on manifolds which vary over time. Our key contribution is the introduction of generation rate, which corresponds to the local deformation of manifold over time around an image component. We show that the generation rate is highly correlated with intuitive visual properties, such as visual saliency, of the image component. Further, we propose an efficient and differentiable scheme to estimate the generation rate for a given image component over time, giving rise to a generation curve. The differentiable nature of our scheme allows us to control the shape of the generation curve via optimization. Using different loss functions, our generation curve matching algorithm provides a unified framework for a range of image manipulation tasks, including semantic transfer, object removal, saliency manipulation, image blending, etc. We conduct comprehensive analytical evaluations to support our findings and evaluate our framework on various manipulation tasks. The results show that our method consistently leads to better manipulation results, compared to recent baselines.