Rayleigh EigenDirections (REDs): GAN latent space traversals for multidimensional features
This work addresses the challenge of fine-grained image manipulation for researchers and practitioners in generative modeling, though it is incremental as it builds on existing latent space traversal techniques.
The paper tackles the problem of controlling multidimensional image features in GAN latent spaces, such as facial identity and object appearance, by introducing Rayleigh EigenDirections (REDs) that enable curved paths to vary one feature set while keeping others constant, achieving capabilities beyond previous methods as demonstrated on StyleGAN2 for faces and living rooms.
We present a method for finding paths in a deep generative model's latent space that can maximally vary one set of image features while holding others constant. Crucially, unlike past traversal approaches, ours can manipulate multidimensional features of an image such as facial identity and pixels within a specified region. Our method is principled and conceptually simple: optimal traversal directions are chosen by maximizing differential changes to one feature set such that changes to another set are negligible. We show that this problem is nearly equivalent to one of Rayleigh quotient maximization, and provide a closed-form solution to it based on solving a generalized eigenvalue equation. We use repeated computations of the corresponding optimal directions, which we call Rayleigh EigenDirections (REDs), to generate appropriately curved paths in latent space. We empirically evaluate our method using StyleGAN2 on two image domains: faces and living rooms. We show that our method is capable of controlling various multidimensional features out of the scope of previous latent space traversal methods: face identity, spatial frequency bands, pixels within a region, and the appearance and position of an object. Our work suggests that a wealth of opportunities lies in the local analysis of the geometry and semantics of latent spaces.