Exploring the latent space of diffusion models directly through singular value decomposition
This work addresses the problem of interpretable and versatile image editing for users of diffusion models, though it is incremental as it builds on existing Stable Diffusion Models.
The paper tackles the challenge of interpreting and controlling the latent space of diffusion models for image editing by applying Singular Value Decomposition (SVD) to discover properties that enable attribute learning from a single pair of latent codes, resulting in a framework that maintains identity fidelity without requiring data collection.
Despite the groundbreaking success of diffusion models in generating high-fidelity images, their latent space remains relatively under-explored, even though it holds significant promise for enabling versatile and interpretable image editing capabilities. The complicated denoising trajectory and high dimensionality of the latent space make it extremely challenging to interpret. Existing methods mainly explore the feature space of U-Net in Diffusion Models (DMs) instead of the latent space itself. In contrast, we directly investigate the latent space via Singular Value Decomposition (SVD) and discover three useful properties that can be used to control generation results without the requirements of data collection and maintain identity fidelity generated images. Based on these properties, we propose a novel image editing framework that is capable of learning arbitrary attributes from one pair of latent codes destined by text prompts in Stable Diffusion Models. To validate our approach, extensive experiments are conducted to demonstrate its effectiveness and flexibility in image editing. We will release our codes soon to foster further research and applications in this area.