Soda-Eval: Open-Domain Dialogue Evaluation in the age of LLMs
This addresses the need for better evaluation benchmarks for contemporary dialogue models, though it is incremental as it builds on existing datasets and methods.
The paper tackles the problem of evaluating open-domain dialogue systems by introducing Soda-Eval, an annotated dataset with over 120K turn-level assessments, and finds that fine-tuning open-access LLMs improves performance over few-shot inference in terms of correlation and explanation.
Although human evaluation remains the gold standard for open-domain dialogue evaluation, the growing popularity of automated evaluation using Large Language Models (LLMs) has also extended to dialogue. However, most frameworks leverage benchmarks that assess older chatbots on aspects such as fluency and relevance, which are not reflective of the challenges associated with contemporary models. In fact, a qualitative analysis on Soda, a GPT-3.5 generated dialogue dataset, suggests that current chatbots may exhibit several recurring issues related to coherence and commonsense knowledge, but generally produce highly fluent and relevant responses. Noting the aforementioned limitations, this paper introduces Soda-Eval, an annotated dataset based on Soda that covers over 120K turn-level assessments across 10K dialogues, where the annotations were generated by GPT-4. Using Soda-Eval as a benchmark, we then study the performance of several open-access instruction-tuned LLMs, finding that dialogue evaluation remains challenging. Fine-tuning these models improves performance over few-shot inferences, both in terms of correlation and explanation.