On The Open Prompt Challenge In Conditional Audio Generation
This addresses the user prompt challenge for commercializing audio generation, but it is incremental as it builds on existing TTA models.
The paper tackles the problem of under-specified user prompts in text-to-audio generation, which leads to poor alignment and audio quality, by rewriting prompts and using text-audio alignment feedback, resulting in marked improvements in both alignment and music audio quality.
Text-to-audio generation (TTA) produces audio from a text description, learning from pairs of audio samples and hand-annotated text. However, commercializing audio generation is challenging as user-input prompts are often under-specified when compared to text descriptions used to train TTA models. In this work, we treat TTA models as a ``blackbox'' and address the user prompt challenge with two key insights: (1) User prompts are generally under-specified, leading to a large alignment gap between user prompts and training prompts. (2) There is a distribution of audio descriptions for which TTA models are better at generating higher quality audio, which we refer to as ``audionese''. To this end, we rewrite prompts with instruction-tuned models and propose utilizing text-audio alignment as feedback signals via margin ranking learning for audio improvements. On both objective and subjective human evaluations, we observed marked improvements in both text-audio alignment and music audio quality.