MORQA: Benchmarking Evaluation Metrics for Medical Open-Ended Question Answering
This addresses the need for better evaluation methods in medical AI applications, where accuracy and relevance are critical, though it is incremental as it builds on existing benchmark and LLM evaluation approaches.
The paper tackled the problem of evaluating natural language generation systems in medical open-ended question answering, where traditional metrics like BLEU and ROUGE are inadequate, by introducing MORQA, a multilingual benchmark with expert-annotated datasets, and found that LLM-based evaluators like GPT-4 significantly outperform traditional metrics in correlating with human judgments.
Evaluating natural language generation (NLG) systems in the medical domain presents unique challenges due to the critical demands for accuracy, relevance, and domain-specific expertise. Traditional automatic evaluation metrics, such as BLEU, ROUGE, and BERTScore, often fall short in distinguishing between high-quality outputs, especially given the open-ended nature of medical question answering (QA) tasks where multiple valid responses may exist. In this work, we introduce MORQA (Medical Open-Response QA), a new multilingual benchmark designed to assess the effectiveness of NLG evaluation metrics across three medical visual and text-based QA datasets in English and Chinese. Unlike prior resources, our datasets feature 2-4+ gold-standard answers authored by medical professionals, along with expert human ratings for three English and Chinese subsets. We benchmark both traditional metrics and large language model (LLM)-based evaluators, such as GPT-4 and Gemini, finding that LLM-based approaches significantly outperform traditional metrics in correlating with expert judgments. We further analyze factors driving this improvement, including LLMs' sensitivity to semantic nuances and robustness to variability among reference answers. Our results provide the first comprehensive, multilingual qualitative study of NLG evaluation in the medical domain, highlighting the need for human-aligned evaluation methods. All datasets and annotations will be publicly released to support future research.