Visual Prompt Tuning for Generative Transfer Learning
This work addresses the problem of domain adaptation for generative image models, offering a method to transfer knowledge from large datasets to new visual domains with varying data sizes, though it is incremental as it builds on existing prompt tuning techniques.
The paper tackles efficient generative image modeling across domains by adapting vision transformers via prompt tuning, achieving significantly better image generation quality than existing methods.
Transferring knowledge from an image synthesis model trained on a large dataset is a promising direction for learning generative image models from various domains efficiently. While previous works have studied GAN models, we present a recipe for learning vision transformers by generative knowledge transfer. We base our framework on state-of-the-art generative vision transformers that represent an image as a sequence of visual tokens to the autoregressive or non-autoregressive transformers. To adapt to a new domain, we employ prompt tuning, which prepends learnable tokens called prompt to the image token sequence, and introduce a new prompt design for our task. We study on a variety of visual domains, including visual task adaptation benchmark~\cite{zhai2019large}, with varying amount of training images, and show effectiveness of knowledge transfer and a significantly better image generation quality over existing works.