Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation
This addresses the issue of reward hacking and unintended side-effects in fine-tuning for text-to-image generation, which is important for developers and users seeking safer and more reliable AI-generated images, though it is an incremental improvement over existing reward fine-tuning methods.
The paper tackles the problem of fine-tuning text-to-image models to improve specific quality aspects like safety and plausibility without degrading others, by proposing Focus-N-Fix, a region-aware method that corrects only problematic image regions, resulting in significant improvements in localized criteria with minimal degradation elsewhere.
Text-to-image (T2I) generation has made significant advances in recent years, but challenges still remain in the generation of perceptual artifacts, misalignment with complex prompts, and safety. The prevailing approach to address these issues involves collecting human feedback on generated images, training reward models to estimate human feedback, and then fine-tuning T2I models based on the reward models to align them with human preferences. However, while existing reward fine-tuning methods can produce images with higher rewards, they may change model behavior in unexpected ways. For example, fine-tuning for one quality aspect (e.g., safety) may degrade other aspects (e.g., prompt alignment), or may lead to reward hacking (e.g., finding a way to increase rewards without having the intended effect). In this paper, we propose Focus-N-Fix, a region-aware fine-tuning method that trains models to correct only previously problematic image regions. The resulting fine-tuned model generates images with the same high-level structure as the original model but shows significant improvements in regions where the original model was deficient in safety (over-sexualization and violence), plausibility, or other criteria. Our experiments demonstrate that Focus-N-Fix improves these localized quality aspects with little or no degradation to others and typically imperceptible changes in the rest of the image. Disclaimer: This paper contains images that may be overly sexual, violent, offensive, or harmful.