Universal Few-Shot Spatial Control for Diffusion Models
This addresses the need for efficient and adaptable spatial control in text-to-image generation, offering a versatile solution for novel tasks with minimal data, though it is incremental in improving existing control adapters.
The paper tackles the problem of limited adaptability and high training costs in spatial control for diffusion models by proposing Universal Few-Shot Control (UFC), which achieves fine-grained control with only 30 annotated examples per novel task and competitive performance using 0.1% of full training data.
Spatial conditioning in pretrained text-to-image diffusion models has significantly improved fine-grained control over the structure of generated images. However, existing control adapters exhibit limited adaptability and incur high training costs when encountering novel spatial control conditions that differ substantially from the training tasks. To address this limitation, we propose Universal Few-Shot Control (UFC), a versatile few-shot control adapter capable of generalizing to novel spatial conditions. Given a few image-condition pairs of an unseen task and a query condition, UFC leverages the analogy between query and support conditions to construct task-specific control features, instantiated by a matching mechanism and an update on a small set of task-specific parameters. Experiments on six novel spatial control tasks show that UFC, fine-tuned with only 30 annotated examples of novel tasks, achieves fine-grained control consistent with the spatial conditions. Notably, when fine-tuned with 0.1% of the full training data, UFC achieves competitive performance with the fully supervised baselines in various control tasks. We also show that UFC is applicable agnostically to various diffusion backbones and demonstrate its effectiveness on both UNet and DiT architectures. Code is available at https://github.com/kietngt00/UFC.