Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage
This work addresses the problem of handling limited data in healthcare applications, specifically for automating patient message urgency classification, though it is incremental in adapting existing methods.
The paper tackled the challenge of training deep learning classifiers on small, imbalanced healthcare datasets by proposing a novel framework based on BioBERT, which improved performance by 4.3% in macro F1 score for patient message triage.
Small and imbalanced datasets commonly seen in healthcare represent a challenge when training classifiers based on deep learning models. So motivated, we propose a novel framework based on BioBERT (Bidirectional Encoder Representations from Transformers forBiomedical TextMining). Specifically, (i) we introduce Label Embeddings for Self-Attention in each layer of BERT, which we call LESA-BERT, and (ii) by distilling LESA-BERT to smaller variants, we aim to reduce overfitting and model size when working on small datasets. As an application, our framework is utilized to build a model for patient portal message triage that classifies the urgency of a message into three categories: non-urgent, medium and urgent. Experiments demonstrate that our approach can outperform several strong baseline classifiers by a significant margin of 4.3% in terms of macro F1 score. The code for this project is publicly available at \url{https://github.com/shijing001/text_classifiers}.