LyricJam: A system for generating lyrics for live instrumental music
This addresses the challenge of real-time lyric generation for musicians during jam sessions, offering a creative tool that is incremental in improving alignment techniques.
The paper tackles the problem of generating lyrics in real-time for live instrumental music by proposing two novel alignment methods for audio and text latent spaces, resulting in a system that users preferred over a baseline model and found useful for composition and improvisation.
We describe a real-time system that receives a live audio stream from a jam session and generates lyric lines that are congruent with the live music being played. Two novel approaches are proposed to align the learned latent spaces of audio and text representations that allow the system to generate novel lyric lines matching live instrumental music. One approach is based on adversarial alignment of latent representations of audio and lyrics, while the other approach learns to transfer the topology from the music latent space to the lyric latent space. A user study with music artists using the system showed that the system was useful not only in lyric composition, but also encouraged the artists to improvise and find new musical expressions. Another user study demonstrated that users preferred the lines generated using the proposed methods to the lines generated by a baseline model.