On the diminishing return of labeling clinical reports
This work addresses the problem of data efficiency in medical NLP for clinicians and researchers, showing incremental insights by challenging the assumption that larger datasets always lead to better performance in this domain.
The study found that for medical NLP tasks like abnormality classification in chest x-ray reports, high-performing models can be achieved with small labeled datasets, contrary to trends in non-medical domains, and these models significantly outperform current state-of-the-art rule-based systems.
Ample evidence suggests that better machine learning models may be steadily obtained by training on increasingly larger datasets on natural language processing (NLP) problems from non-medical domains. Whether the same holds true for medical NLP has by far not been thoroughly investigated. This work shows that this is indeed not always the case. We reveal the somehow counter-intuitive observation that performant medical NLP models may be obtained with small amount of labeled data, quite the opposite to the common belief, most likely due to the domain specificity of the problem. We show quantitatively the effect of training data size on a fixed test set composed of two of the largest public chest x-ray radiology report datasets on the task of abnormality classification. The trained models not only make use of the training data efficiently, but also outperform the current state-of-the-art rule-based systems by a significant margin.