Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores
This work addresses the need for a standardized benchmark to assess multimodal reasoning in music AI, which is incremental as it builds on existing evaluation frameworks but focuses on a novel domain.
The authors tackled the problem of evaluating large language and vision-language models' ability to understand complete musical scores, introducing the Musical Score Understanding Benchmark (MSU-Bench) with 1,800 QA pairs across textual and visual modalities, and found that fine-tuning substantially improves results while revealing modality gaps and performance instability.
Understanding complete musical scores entails integrated reasoning over pitch, rhythm, harmony, and large-scale structure, yet the ability of Large Language Models and Vision-Language Models to interpret full musical notation remains insufficiently examined. We introduce the Musical Score Understanding Benchmark (MSU-Bench), the first large-scale, human-curated benchmark for score-level musical understanding across textual (ABC notation) and visual (PDF) modalities. MSU-Bench contains 1,800 generative Question-Answering pairs from works by Bach, Beethoven, Chopin, Debussy, and others, organised into four levels of increasing difficulty, ranging from onset information to texture and form. Evaluations of more than fifteen state-of-the-art models, in both zero-shot and fine-tuned settings, reveal pronounced modality gaps, unstable level-wise performance, and challenges in maintaining multilevel correctness. Fine-tuning substantially improves results across modalities while preserving general knowledge, positioning MSU-Bench as a robust foundation for future research in multimodal reasoning. To facilitate further research, we publicly release MSU-Bench and all associated resources.