The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models
This provides a standardized evaluation framework for assessing LLMs' music-related capabilities, addressing a gap in existing benchmarks, though it is incremental as it applies existing benchmark methods to a new domain.
The authors tackled the lack of a dedicated benchmark for evaluating large language models' musical abilities by introducing ZIQI-Eval, a comprehensive music benchmark with over 14,000 data entries across 10 categories, and found that all 16 tested LLMs performed poorly, indicating significant room for improvement.
Benchmark plays a pivotal role in assessing the advancements of large language models (LLMs). While numerous benchmarks have been proposed to evaluate LLMs' capabilities, there is a notable absence of a dedicated benchmark for assessing their musical abilities. To address this gap, we present ZIQI-Eval, a comprehensive and large-scale music benchmark specifically designed to evaluate the music-related capabilities of LLMs. ZIQI-Eval encompasses a wide range of questions, covering 10 major categories and 56 subcategories, resulting in over 14,000 meticulously curated data entries. By leveraging ZIQI-Eval, we conduct a comprehensive evaluation over 16 LLMs to evaluate and analyze LLMs' performance in the domain of music. Results indicate that all LLMs perform poorly on the ZIQI-Eval benchmark, suggesting significant room for improvement in their musical capabilities. With ZIQI-Eval, we aim to provide a standardized and robust evaluation framework that facilitates a comprehensive assessment of LLMs' music-related abilities. The dataset is available at GitHub\footnote{https://github.com/zcli-charlie/ZIQI-Eval} and HuggingFace\footnote{https://huggingface.co/datasets/MYTH-Lab/ZIQI-Eval}.