LLM-RadJudge: Achieving Radiologist-Level Evaluation for X-Ray Report Generation
This provides an accessible and efficient evaluation method for radiology report generation, facilitating more clinically relevant AI development, though it is incremental in improving existing evaluation metrics.
The study tackled the problem of evaluating radiology reports by proposing a novel framework using large language models (LLMs), achieving evaluation consistency close to radiologists with GPT-4 and distilling it into a smaller model with comparable capabilities.
Evaluating generated radiology reports is crucial for the development of radiology AI, but existing metrics fail to reflect the task's clinical requirements. This study proposes a novel evaluation framework using large language models (LLMs) to compare radiology reports for assessment. We compare the performance of various LLMs and demonstrate that, when using GPT-4, our proposed metric achieves evaluation consistency close to that of radiologists. Furthermore, to reduce costs and improve accessibility, making this method practical, we construct a dataset using LLM evaluation results and perform knowledge distillation to train a smaller model. The distilled model achieves evaluation capabilities comparable to GPT-4. Our framework and distilled model offer an accessible and efficient evaluation method for radiology report generation, facilitating the development of more clinically relevant models. The model will be further open-sourced and accessible.