A Human Subject Study of Named Entity Recognition (NER) in Conversational Music Recommendation Queries
This work addresses the problem of improving NER for conversational music recommendation systems, though it is incremental as it focuses on evaluation and error analysis rather than proposing a new method.
The study tackled named entity recognition in noisy conversational music recommendation queries, finding that both humans and fine-tuned transformer models struggled under strict evaluation, with humans achieving higher precision and models higher recall due to pre-training exposure.
We conducted a human subject study of named entity recognition on a noisy corpus of conversational music recommendation queries, with many irregular and novel named entities. We evaluated the human NER linguistic behaviour in these challenging conditions and compared it with the most common NER systems nowadays, fine-tuned transformers. Our goal was to learn about the task to guide the design of better evaluation methods and NER algorithms. The results showed that NER in our context was quite hard for both human and algorithms under a strict evaluation schema; humans had higher precision, while the model higher recall because of entity exposure especially during pre-training; and entity types had different error patterns (e.g. frequent typing errors for artists). The released corpus goes beyond predefined frames of interaction and can support future work in conversational music recommendation.