Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models
This addresses a practical problem for developers fine-tuning text-to-image models with human feedback, though it is an incremental improvement over existing reward optimization methods.
The paper tackles reward overoptimization in fine-tuning text-to-image models by showing that poorly aligned reward models cause this issue, and proposes TextNorm, a confidence-calibration method that reduces overoptimization and achieves twice as many wins in human evaluation for text-image alignment compared to baselines.
Fine-tuning text-to-image models with reward functions trained on human feedback data has proven effective for aligning model behavior with human intent. However, excessive optimization with such reward models, which serve as mere proxy objectives, can compromise the performance of fine-tuned models, a phenomenon known as reward overoptimization. To investigate this issue in depth, we introduce the Text-Image Alignment Assessment (TIA2) benchmark, which comprises a diverse collection of text prompts, images, and human annotations. Our evaluation of several state-of-the-art reward models on this benchmark reveals their frequent misalignment with human assessment. We empirically demonstrate that overoptimization occurs notably when a poorly aligned reward model is used as the fine-tuning objective. To address this, we propose TextNorm, a simple method that enhances alignment based on a measure of reward model confidence estimated across a set of semantically contrastive text prompts. We demonstrate that incorporating the confidence-calibrated rewards in fine-tuning effectively reduces overoptimization, resulting in twice as many wins in human evaluation for text-image alignment compared against the baseline reward models.