A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models
This work addresses the need for better evaluation of LLMs in TCM for researchers and practitioners, though it is incremental as it builds on existing benchmarks by adding treatment decision-making assessment.
The authors tackled the problem of evaluating large language models (LLMs) in Traditional Chinese Medicine (TCM) by creating a comprehensive benchmark dataset called TCM-BEST4SDT, which includes four tasks and was tested on 15 mainstream LLMs to assess clinical application capabilities.
The emergence of Large Language Models (LLMs) within the Traditional Chinese Medicine (TCM) domain presents an urgent need to assess their clinical application capabilities. However, such evaluations are challenged by the individualized, holistic, and diverse nature of TCM's "Syndrome Differentiation and Treatment" (SDT). Existing benchmarks are confined to knowledge-based question-answering or the accuracy of syndrome differentiation, often neglecting assessment of treatment decision-making. Here, we propose a comprehensive, clinical case-based benchmark spearheaded by TCM experts, and a specialized reward model employed to quantify prescription-syndrome congruence. Data annotation follows a rigorous pipeline. This benchmark, designated TCM-BEST4SDT, encompasses four tasks, including TCM Basic Knowledge, Medical Ethics, LLM Content Safety, and SDT. The evaluation framework integrates three mechanisms, namely selected-response evaluation, judge model evaluation, and reward model evaluation. The effectiveness of TCM-BEST4SDT was corroborated through experiments on 15 mainstream LLMs, spanning both general and TCM domains. To foster the development of intelligent TCM research, TCM-BEST4SDT is now publicly available.