Cover Song Synthesis by Analogy
This addresses a specific challenge in music style transfer for audio synthesis, but it is incremental as it builds on existing techniques like NMF and audio mosaicing.
The paper tackles the problem of synthesizing cover songs by analogy, enabling the generation of a cover of a song in the style of a different artist using an example cover pair, and demonstrates the approach with examples like creating a Michael Jackson cover by Alien Ant Farm.
In this work, we pose and address the following "cover song analogies" problem: given a song A by artist 1 and a cover song A' of this song by artist 2, and given a different song B by artist 1, synthesize a song B' which is a cover of B in the style of artist 2. Normally, such a polyphonic style transfer problem would be quite challenging, but we show how the cover songs example constrains the problem, making it easier to solve. First, we extract the longest common beat-synchronous subsequence between A and A', and we time stretch the corresponding beat intervals in A' so that they align with A. We then derive a version of joint 2D convolutional NMF, which we apply to the constant-Q spectrograms of the synchronized segments to learn a translation dictionary of sound templates from A to A'. Finally, we apply the learned templates as filters to the song B, and we mash up the translated filtered components into the synthesized song B' using audio mosaicing. We showcase our algorithm on several examples, including a synthesized cover version of Michael Jackson's "Bad" by Alien Ant Farm, learned from the latter's "Smooth Criminal" cover.'