Automatic Realistic Music Video Generation from Segments of Youtube Videos
This addresses the need for automated video creation tools for content creators, though it is incremental as it builds on existing segmentation and clustering techniques.
The paper tackles the problem of automatically generating realistic music videos from YouTube segments by analyzing input music for genre and color histograms, then assembling clips around music boundaries, achieving results where 45% of generated videos were mistaken for professional ones and 21.6% for amateur-made in user tests.
A Music Video (MV) is a video aiming at visually illustrating or extending the meaning of its background music. This paper proposes a novel method to automatically generate, from an input music track, a music video made of segments of Youtube music videos which would fit this music. The system analyzes the input music to find its genre (pop, rock, ...) and finds segmented MVs with the same genre in the database. Then, a K-Means clustering is done to group video segments by color histogram, meaning segments of MVs having the same global distribution of colors. A few clusters are randomly selected, then are assembled around music boundaries, which are moments where a significant change in the music occurs (for instance, transitioning from verse to chorus). This way, when the music changes, the video color mood changes as well. This work aims at generating high-quality realistic MVs, which could be mistaken for man-made MVs. By asking users to identify, in a batch of music videos containing professional MVs, amateur-made MVs and generated MVs by our algorithm, we show that our algorithm gives satisfying results, as 45% of generated videos are mistaken for professional MVs and 21.6% are mistaken for amateur-made MVs. More information can be found in the project website: http://ml.cs.tsinghua.edu.cn/~sarah/