Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces
This work addresses the challenge of improving co-creation in music for composers and listeners, but it is incremental as it builds on existing ML and HCI research.
The study tackled the problem of empowering creators in music generation by evaluating both generative models and steering interfaces through a common framework, finding that more expressive models and steerable interfaces are important and complementary for enhancing communication and creative empowerment, as evidenced by 26 composers creating 100+ pieces and 1000+ listener comparisons.
There is an increasing interest from ML and HCI communities in empowering creators with better generative models and more intuitive interfaces with which to control them. In music, ML researchers have focused on training models capable of generating pieces with increasing long-range structure and musical coherence, while HCI researchers have separately focused on designing steering interfaces that support user control and ownership. In this study, we investigate through a common framework how developments in both models and user interfaces are important for empowering co-creation where the goal is to create music that communicates particular imagery or ideas (e.g., as is common for other purposeful tasks in music creation like establishing mood or creating accompanying music for another media). Our study is distinguished in that it measures communication through both composer's self-reported experiences, and how listeners evaluate this communication through the music. In an evaluation study with 26 composers creating 100+ pieces of music and listeners providing 1000+ head-to-head comparisons, we find that more expressive models and more steerable interfaces are important and complementary ways to make a difference in composers communicating through music and supporting their creative empowerment.