DualVAE: Controlling Colours of Generated and Real Images
This addresses a specific need for artists and graphic designers by enabling color-controlled image generation and recoloring, representing an incremental improvement over existing VQ-VAE methods.
The paper tackled the problem of controlling color attributes in image generation and manipulation by introducing DualVAE, a hybrid model that learns disentangled representations for color and geometry, resulting in generated images with FID nearly two times better than VQ-GAN on various datasets.
Colour controlled image generation and manipulation are of interest to artists and graphic designers. Vector Quantised Variational AutoEncoders (VQ-VAEs) with autoregressive (AR) prior are able to produce high quality images, but lack an explicit representation mechanism to control colour attributes. We introduce DualVAE, a hybrid representation model that provides such control by learning disentangled representations for colour and geometry. The geometry is represented by an image intensity mapping that identifies structural features. The disentangled representation is obtained by two novel mechanisms: (i) a dual branch architecture that separates image colour attributes from geometric attributes, and (ii) a new ELBO that trains the combined colour and geometry representations. DualVAE can control the colour of generated images, and recolour existing images by transferring the colour latent representation obtained from an exemplar image. We demonstrate that DualVAE generates images with FID nearly two times better than VQ-GAN on a diverse collection of datasets, including animated faces, logos and artistic landscapes.