MedReadMe: A Systematic Study for Fine-grained Sentence Readability in Medical Domain
This work addresses the challenge of making medical texts more accessible by providing a systematic approach to fine-grained readability assessment, though it is incremental in improving existing metrics.
The paper tackled the problem of measuring readability in medical texts by introducing the MedReadMe dataset with 4,520 annotated sentences and benchmarking readability metrics, finding that adding a jargon span feature improved correlation with human judgments.
Medical texts are notoriously challenging to read. Properly measuring their readability is the first step towards making them more accessible. In this paper, we present a systematic study on fine-grained readability measurements in the medical domain at both sentence-level and span-level. We introduce a new dataset MedReadMe, which consists of manually annotated readability ratings and fine-grained complex span annotation for 4,520 sentences, featuring two novel "Google-Easy" and "Google-Hard" categories. It supports our quantitative analysis, which covers 650 linguistic features and automatic complex word and jargon identification. Enabled by our high-quality annotation, we benchmark and improve several state-of-the-art sentence-level readability metrics for the medical domain specifically, which include unsupervised, supervised, and prompting-based methods using recently developed large language models (LLMs). Informed by our fine-grained complex span annotation, we find that adding a single feature, capturing the number of jargon spans, into existing readability formulas can significantly improve their correlation with human judgments. The data is available at tinyurl.com/medreadme-repo