Cost-Aware Routing for Efficient Text-To-Image Generation
This work addresses efficiency challenges in text-to-image generation for users needing faster or cheaper image synthesis, though it is incremental as it builds on existing models without introducing new generation paradigms.
The paper tackles the high computational cost of diffusion models in text-to-image generation by proposing a cost-aware routing framework that dynamically allocates prompts to different generation functions based on complexity, achieving higher average quality than any single model alone on COCO and DiffusionDB benchmarks.
Diffusion models are well known for their ability to generate a high-fidelity image for an input prompt through an iterative denoising process. Unfortunately, the high fidelity also comes at a high computational cost due the inherently sequential generative process. In this work, we seek to optimally balance quality and computational cost, and propose a framework to allow the amount of computation to vary for each prompt, depending on its complexity. Each prompt is automatically routed to the most appropriate text-to-image generation function, which may correspond to a distinct number of denoising steps of a diffusion model, or a disparate, independent text-to-image model. Unlike uniform cost reduction techniques (e.g., distillation, model quantization), our approach achieves the optimal trade-off by learning to reserve expensive choices (e.g., 100+ denoising steps) only for a few complex prompts, and employ more economical choices (e.g., small distilled model) for less sophisticated prompts. We empirically demonstrate on COCO and DiffusionDB that by learning to route to nine already-trained text-to-image models, our approach is able to deliver an average quality that is higher than that achievable by any of these models alone.