Resonate: Reinforcing Text-to-Audio Generation via Online Feedback from Large Audio Language Models
This work addresses the challenge of generating high-quality and semantically aligned audio from text, which is incremental as it applies existing RL methods to an under-explored domain.
The paper tackled the problem of improving text-to-audio generation by integrating online reinforcement learning with rewards from large audio language models, resulting in a model that sets a new state-of-the-art on TTA-Bench for audio quality and semantic alignment with only 470M parameters.
Reinforcement Learning (RL) has become an effective paradigm for enhancing Large Language Models (LLMs) and visual generative models. However, its application in text-to-audio (TTA) generation remains largely under-explored. Prior work typically employs offline methods like Direct Preference Optimization (DPO) and leverages Contrastive Language-Audio Pretraining (CLAP) models as reward functions. In this study, we investigate the integration of online Group Relative Policy Optimization (GRPO) into TTA generation. We adapt the algorithm for Flow Matching-based audio models and demonstrate that online RL significantly outperforms its offline counterparts. Furthermore, we incorporate rewards derived from Large Audio Language Models (LALMs), which can provide fine-grained scoring signals that are better aligned with human perception. With only 470M parameters, our final model, \textbf{Resonate}, establishes a new SOTA on TTA-Bench in terms of both audio quality and semantic alignment.