FALL-E: A Foley Sound Synthesis Model and Strategies
This work addresses foley sound synthesis for audio generation applications, representing an incremental improvement in a specific domain.
The paper tackles foley sound synthesis by introducing FALL-E, a system that uses a cascaded model with text conditioning and external language models, achieving second place overall in the DCASE 2023 challenge Task 7, with best diversity, second-best audio quality, and third-best class fitness scores.
This paper introduces FALL-E, a foley synthesis system and its training/inference strategies. The FALL-E model employs a cascaded approach comprising low-resolution spectrogram generation, spectrogram super-resolution, and a vocoder. We trained every sound-related model from scratch using our extensive datasets, and utilized a pre-trained language model. We conditioned the model with dataset-specific texts, enabling it to learn sound quality and recording environment based on text input. Moreover, we leveraged external language models to improve text descriptions of our datasets and performed prompt engineering for quality, coherence, and diversity. FALL-E was evaluated by an objective measure as well as listening tests in the DCASE 2023 challenge Task 7. The submission achieved the second place on average, while achieving the best score for diversity, second place for audio quality, and third place for class fitness.