Generation of lyrics lines conditioned on music audio clips
This provides a creativity tool for songwriters by enabling lyric generation based on audio input.
The authors tackled the problem of generating lyrics lines conditioned on music audio clips by developing a bimodal neural network model, which effectively produced lines with emotional impact matching the audio as shown in evaluations.
We present a system for generating novel lyrics lines conditioned on music audio. A bimodal neural network model learns to generate lines conditioned on any given short audio clip. The model consists of a spectrogram variational autoencoder (VAE) and a text VAE. Both automatic and human evaluations demonstrate effectiveness of our model in generating lines that have an emotional impact matching a given audio clip. The system is intended to serve as a creativity tool for songwriters.