Automatic Assignment of Radiology Examination Protocols Using Pre-trained Language Models with Knowledge Distillation
This work addresses a domain-specific problem for radiology by automating protocol assignment, but it is incremental as it builds on existing BERT models with minor adaptations like knowledge distillation.
The paper tackled the problem of automatically assigning radiology examination protocols for computer tomography, which is repetitive and time-consuming, by developing a deep learning approach using a pre-trained domain-specific BERT model with knowledge distillation, achieving an F1 score of 0.66 and outperforming baseline models like SVM, GBM, RF, and standard BERT.
Selecting radiology examination protocol is a repetitive, and time-consuming process. In this paper, we present a deep learning approach to automatically assign protocols to computer tomography examinations, by pre-training a domain-specific BERT model ($BERT_{rad}$). To handle the high data imbalance across exam protocols, we used a knowledge distillation approach that up-sampled the minority classes through data augmentation. We compared classification performance of the described approach with the statistical n-gram models using Support Vector Machine (SVM), Gradient Boosting Machine (GBM), and Random Forest (RF) classifiers, as well as the Google's $BERT_{base}$ model. SVM, GBM and RF achieved macro-averaged F1 scores of 0.45, 0.45, and 0.6 while $BERT_{base}$ and $BERT_{rad}$ achieved 0.61 and 0.63. Knowledge distillation improved overall performance on the minority classes, achieving a F1 score of 0.66.