Guidance Matters: Rethinking the Evaluation Pitfall for Text-to-Image Generation
This work addresses a critical evaluation flaw for researchers and developers working on text-to-image generation, potentially preventing misdirection in future research by highlighting that many recent methods offer little to no real improvement over basic CFG.
This paper reveals a critical evaluation pitfall in text-to-image generation where human preference models are biased towards large guidance scales, leading to inflated scores for methods that simply increase classifier-free guidance (CFG) scale, even when image quality degrades. The authors introduce a guidance-aware evaluation (GA-Eval) framework to enable fair comparisons and demonstrate that simply increasing CFG scales can compete with most studied diffusion guidance methods, which often suffer from winning rate degradation over standard CFG.
Classifier-free guidance (CFG) has helped diffusion models achieve great conditional generation in various fields. Recently, more diffusion guidance methods have emerged with improved generation quality and human preference. However, can these emerging diffusion guidance methods really achieve solid and significant improvements? In this paper, we rethink recent progress on diffusion guidance. Our work mainly consists of four contributions. First, we reveal a critical evaluation pitfall that common human preference models exhibit a strong bias towards large guidance scales. Simply increasing the CFG scale can easily improve quantitative evaluation scores due to strong semantic alignment, even if image quality is severely damaged (e.g., oversaturation and artifacts). Second, we introduce a novel guidance-aware evaluation (GA-Eval) framework that employs effective guidance scale calibration to enable fair comparison between current guidance methods and CFG by identifying the effects orthogonal and parallel to CFG effects. Third, motivated by the evaluation pitfall, we design Transcendent Diffusion Guidance (TDG) method that can significantly improve human preference scores in the conventional evaluation framework but actually does not work in practice. Fourth, in extensive experiments, we empirically evaluate recent eight diffusion guidance methods within the conventional evaluation framework and the proposed GA-Eval framework. Notably, simply increasing the CFG scales can compete with most studied diffusion guidance methods, while all methods suffer severely from winning rate degradation over standard CFG. Our work would strongly motivate the community to rethink the evaluation paradigm and future directions of this field.