Enhancing disease detection in radiology reports through fine-tuning lightweight LLM on weak labels
This addresses the challenge of limited labeled data for medical AI applications, though it is incremental as it builds on existing fine-tuning and synthetic label methods.
The paper tackles the problem of applying large language models to medical domains by fine-tuning a lightweight LLM (Llama 3.1-8B) on synthetic labels for disease detection in radiology reports, achieving a micro F1 score of 0.91 with high-quality labels and surpassing noisy teacher labels (0.67 vs. 0.63) with low-quality labels.
Despite significant progress in applying large language models (LLMs) to the medical domain, several limitations still prevent them from practical applications. Among these are the constraints on model size and the lack of cohort-specific labeled datasets. In this work, we investigated the potential of improving a lightweight LLM, such as Llama 3.1-8B, through fine-tuning with datasets using synthetic labels. Two tasks are jointly trained by combining their respective instruction datasets. When the quality of the task-specific synthetic labels is relatively high (e.g., generated by GPT4- o), Llama 3.1-8B achieves satisfactory performance on the open-ended disease detection task, with a micro F1 score of 0.91. Conversely, when the quality of the task-relevant synthetic labels is relatively low (e.g., from the MIMIC-CXR dataset), fine-tuned Llama 3.1-8B is able to surpass its noisy teacher labels (micro F1 score of 0.67 v.s. 0.63) when calibrated against curated labels, indicating the strong inherent underlying capability of the model. These findings demonstrate the potential of fine-tuning LLMs with synthetic labels, offering a promising direction for future research on LLM specialization in the medical domain.