Scale Where It Matters: Training-Free Localized Scaling for Diffusion Models
This work addresses a key efficiency bottleneck in text-to-image generation for users of diffusion models, offering a novel training-free approach to reduce computational costs without sacrificing quality.
The paper tackles the problem of inefficient full-image scaling in diffusion models by introducing localized test-time scaling, which adaptively resamples only defective regions to improve image quality. The proposed LoTTS framework achieves state-of-the-art performance, enhancing local quality and global fidelity while reducing GPU cost by 2-4x compared to Best-of-N sampling.
Diffusion models have become the dominant paradigm in text-to-image generation, and test-time scaling (TTS) further improves quality by allocating more computation during inference. However, existing TTS methods operate at the full-image level, overlooking the fact that image quality is often spatially heterogeneous. This leads to unnecessary computation on already satisfactory regions and insufficient correction of localized defects. In this paper, we explore a new direction - Localized TTS - that adaptively resamples defective regions while preserving high-quality regions, thereby substantially reducing the search space. This paradigm poses two central challenges: accurately localizing defects and maintaining global consistency. We propose LoTTS, the first fully training-free framework for localized TTS. For defect localization, LoTTS contrasts cross- and self-attention signals under quality-aware prompts (e.g., high-quality vs. low-quality) to identify defective regions, and then refines them into coherent masks. For consistency, LoTTS perturbs only defective regions and denoises them locally, ensuring that corrections remain confined while the rest of the image remains undisturbed. Extensive experiments on SD2.1, SDXL, and FLUX demonstrate that LoTTS achieves state-of-the-art performance: it consistently improves both local quality and global fidelity, while reducing GPU cost by 2-4x compared to Best-of-N sampling. These findings establish localized TTS as a promising new direction for scaling diffusion models at inference time.