Highly Compressed Tokenizer Can Generate Without Training
This provides a novel, training-free method for image editing and generation, which could benefit researchers and practitioners in computer vision by reducing computational costs.
The paper tackles the problem of image editing and generation by using a highly compressed 1D tokenizer, showing that heuristic manipulations like copying tokens enable fine-grained editing, and gradient-based optimization generates realistic samples without training a generative model, achieving results with as few as 32 tokens.
Commonly used image tokenizers produce a 2D grid of spatially arranged tokens. In contrast, so-called 1D image tokenizers represent images as highly compressed one-dimensional sequences of as few as 32 discrete tokens. We find that the high degree of compression achieved by a 1D tokenizer with vector quantization enables image editing and generative capabilities through heuristic manipulation of tokens, demonstrating that even very crude manipulations -- such as copying and replacing tokens between latent representations of images -- enable fine-grained image editing by transferring appearance and semantic attributes. Motivated by the expressivity of the 1D tokenizer's latent space, we construct an image generation pipeline leveraging gradient-based test-time optimization of tokens with plug-and-play loss functions such as reconstruction or CLIP similarity. Our approach is demonstrated for inpainting and text-guided image editing use cases, and can generate diverse and realistic samples without requiring training of any generative model.