Performance Plateaus in Inference-Time Scaling for Text-to-Image Diffusion Without External Models
This addresses efficiency for researchers and practitioners using GPUs with limited VRAM, but it is incremental as it builds on prior work on inference-time scaling.
The paper tackled the problem of performance plateaus in inference-time scaling for text-to-image diffusion models without external models, finding that a relatively small number of optimization steps suffices to achieve maximum performance across datasets and backbones.
Recently, it has been shown that investing computing resources in searching for good initial noise for a text-to-image diffusion model helps improve performance. However, previous studies required external models to evaluate the resulting images, which is impossible on GPUs with small VRAM. For these reasons, we apply Best-of-N inference-time scaling to algorithms that optimize the initial noise of a diffusion model without external models across multiple datasets and backbones. We demonstrate that inference-time scaling for text-to-image diffusion models in this setting quickly reaches a performance plateau, and a relatively small number of optimization steps suffices to achieve the maximum achievable performance with each algorithm.