Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning
This addresses the problem of building AI musicians by providing synthetic data for sheet music interpretation, though it is incremental as it builds on existing music theory and data synthesis methods.
The paper tackled the lack of evaluation benchmarks and training data for sheet music reasoning in AI by introducing a framework that synthesizes verifiable sheet music problems using music theory rules, resulting in the SSMR-Bench and training set, which improved models on benchmarks like MusicTheoryBench and MMMU.
Enhancing the ability of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) to interpret sheet music is a crucial step toward building AI musicians. However, current research lacks both evaluation benchmarks and training data for sheet music reasoning. Inspired by mathematics, where simple operations yield infinite verifiable problems, we introduce a novel approach that treats core music theory rules, such as those governing beats and intervals, as programmatic functions to systematically synthesize a vast and diverse corpus of sheet music reasoning problems. This approach allows us to introduce a data synthesis framework that generates verifiable sheet music questions in both textual and visual modalities, leading to the Synthetic Sheet Music Reasoning Benchmark (SSMR-Bench) and a complementary training set. Evaluation results on SSMR-Bench highlight the key role reasoning plays in interpreting sheet music, while also pointing out the ongoing challenges in understanding sheet music in a visual format. By leveraging synthetic data for RLVR, all models show significant improvements on the SSMR-Bench. Additionally, they also demonstrate considerable advancements on previously established human-crafted benchmarks, such as MusicTheoryBench and the music subset of MMMU. Finally, our results show that the enhanced reasoning ability can also facilitate music composition.