The Role of Large Language Models in Musicology: Are We Ready to Trust the Machines?
This work addresses the trustworthiness of LLMs for musicology researchers and students, but it is incremental as it builds on existing retrieval-augmented generation methods.
The paper tackled the problem of assessing the reliability of Large Language Models (LLMs) in musicology by proposing a semi-automatic method to create a benchmark using retrieval-augmented generation, and found that current vanilla LLMs are less reliable than retrieval-augmented models from music dictionaries, with evaluation on 400 human-validated questions.
In this work, we explore the use and reliability of Large Language Models (LLMs) in musicology. From a discussion with experts and students, we assess the current acceptance and concerns regarding this, nowadays ubiquitous, technology. We aim to go one step further, proposing a semi-automatic method to create an initial benchmark using retrieval-augmented generation models and multiple-choice question generation, validated by human experts. Our evaluation on 400 human-validated questions shows that current vanilla LLMs are less reliable than retrieval augmented generation from music dictionaries. This paper suggests that the potential of LLMs in musicology requires musicology driven research that can specialized LLMs by including accurate and reliable domain knowledge.