Shimon the Robot Film Composer and DeepScore: An LSTM for Generation of Film Scores based on Visual Analysis
This addresses film music composition for filmmakers and composers, but it appears incremental as it builds on existing visual analysis technologies.
The researchers tackled film music generation by extracting visual features like character emotions and director aesthetic choices to understand narrative arcs, then developed two systems: a robotic composer for real-time performance and a software system for narrative-driven scores.
Composing for a film requires developing an understanding of the film, its characters and the film aesthetic choices made by the director. We propose using existing visual analysis systems as a core technology for film music generation. We extract film features including main characters and their emotions to develop a computer understanding of the film's narrative arc. This arc is combined with visually analyzed director aesthetic choices including pacing and levels of movement. Two systems are presented, the first using a robotic film composer and marimbist to generate film scores in real-time performance. The second software-based system builds on the results from the robot film composer to create narrative driven film scores.