IRIS: Intrinsic Reward Image Synthesis
This addresses a bottleneck in text-to-image generation for AI researchers by enabling reinforcement learning without external data, though it is incremental as it builds on existing autoregressive models.
The paper tackles the problem of limited human preference data in autoregressive text-to-image generation by proposing IRIS, a framework that uses intrinsic rewards based on maximizing self-uncertainty, achieving performance competitive with or superior to external rewards.
Despite the success of Reinforcement Learning from Human Feedback (RLHF) in language reasoning, its application to autoregressive Text-to-Image (T2I) generation is often constrained by the limited availability of human preference data. This paper explores how an autoregressive T2I model can learn from internal signals without relying on external rewards or labeled data. Contrary to recent findings in text generation, we show that maximizing self-uncertainty, rather than self-certainty, improves image generation. We observe that this is because autoregressive T2I models with low uncertainty tend to generate simple and uniform images, which are less aligned with human preferences. Based on these observations, we propose IRIS (Intrinsic Reward Image Synthesis), the first framework to improve autoregressive T2I models with reinforcement learning using only an intrinsic reward. Empirical results demonstrate that applying IRIS to autoregressive T2I models achieves performance that is competitive with or superior to external rewards.