Mozualization: Crafting Music and Visual Representation with Multimodal AI
This addresses the need for creative music generation tools for users like music enthusiasts, but it is incremental as it builds on existing multimodal AI concepts.
The authors tackled the problem of generating and editing music from diverse multimodal inputs like keywords, images, and sound clips, resulting in a tool called Mozualization that transforms emotional expressions into songs, with evaluation based on a user study involving nine music enthusiasts.
In this work, we introduce Mozualization, a music generation and editing tool that creates multi-style embedded music by integrating diverse inputs, such as keywords, images, and sound clips (e.g., segments from various pieces of music or even a playful cat's meow). Our work is inspired by the ways people express their emotions -- writing mood-descriptive poems or articles, creating drawings with warm or cool tones, or listening to sad or uplifting music. Building on this concept, we developed a tool that transforms these emotional expressions into a cohesive and expressive song, allowing users to seamlessly incorporate their unique preferences and inspirations. To evaluate the tool and, more importantly, gather insights for its improvement, we conducted a user study involving nine music enthusiasts. The study assessed user experience, engagement, and the impact of interacting with and listening to the generated music.