Classification of Radiological Text in Small and Imbalanced Datasets in a Non-English Language
This research is significant for medical practitioners and researchers working with low-resource, non-English radiological text data, offering insights into effective NLP approaches for this challenging scenario.
The paper addresses the challenge of classifying radiological text in small, imbalanced, non-English datasets. They found that BERT-like models pretrained on radiology reports performed optimally compared to few-shot learning with sentence transformers (SetFit) and prompted large language models (LLMs), with LLMs performing the worst. While no model achieved sufficient accuracy for unsupervised classification, they show potential for data filtering.
Natural language processing (NLP) in the medical domain can underperform in real-world applications involving small datasets in a non-English language with few labeled samples and imbalanced classes. There is yet no consensus on how to approach this problem. We evaluated a set of NLP models including BERT-like transformers, few-shot learning with sentence transformers (SetFit), and prompted large language models (LLM), using three datasets of radiology reports on magnetic resonance images of epilepsy patients in Danish, a low-resource language. Our results indicate that BERT-like models pretrained in the target domain of radiology reports currently offer the optimal performances for this scenario. Notably, the SetFit and LLM models underperformed compared to BERT-like models, with LLM performing the worst. Importantly, none of the models investigated was sufficiently accurate to allow for text classification without any supervision. However, they show potential for data filtering, which could reduce the amount of manual labeling required.