PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain
This work addresses the problem of limited non-English biomedical benchmarks for researchers and developers, though it is incremental as it adapts an existing benchmark.
The authors tackled the lack of a comprehensive Chinese benchmark for evaluating large language models in the medical domain by creating PromptCBLUE, a prompt-tuning benchmark based on CBLUE, and reported results from experiments with 9 Chinese LLMs using various fine-tuning techniques.
Biomedical language understanding benchmarks are the driving forces for artificial intelligence applications with large language model (LLM) back-ends. However, most current benchmarks: (a) are limited to English which makes it challenging to replicate many of the successes in English for other languages, or (b) focus on knowledge probing of LLMs and neglect to evaluate how LLMs apply these knowledge to perform on a wide range of bio-medical tasks, or (c) have become a publicly available corpus and are leaked to LLMs during pre-training. To facilitate the research in medical LLMs, we re-build the Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark into a large scale prompt-tuning benchmark, PromptCBLUE. Our benchmark is a suitable test-bed and an online platform for evaluating Chinese LLMs' multi-task capabilities on a wide range bio-medical tasks including medical entity recognition, medical text classification, medical natural language inference, medical dialogue understanding and medical content/dialogue generation. To establish evaluation on these tasks, we have experimented and report the results with the current 9 Chinese LLMs fine-tuned with differtent fine-tuning techniques.