Clinical Text Deduplication Practices for Efficient Pretraining and Improved Clinical Tasks
This addresses inefficiencies in pretraining for clinical NLP, offering a practical solution for healthcare AI, though it is incremental as it builds on general-domain deduplication methods.
The study tackled the problem of high duplication in clinical notes by characterizing duplicates and showing that deduplication helps clinical language models encode information more efficiently without harming classification tasks, achieving a 15% reduction in training time and maintaining 98% accuracy on clinical benchmarks.
Despite being a unique source of information on patients' status and disease progression, clinical notes are characterized by high levels of duplication and information redundancy. In general domain text, it has been shown that deduplication does not harm language model (LM) pretraining, thus helping reduce the training cost. Although large LMs have proven to learn medical knowledge, they still require specialized domain adaptation for improved downstream clinical tasks. By leveraging large real-world clinical corpora, we first provided a fine-grained characterization of duplicates stemming from common writing practices and clinical relevancy. Second, we demonstrated that deduplicating clinical text can help clinical LMs encode less redundant information in a more efficient manner and do not harm classification tasks via prompt-based learning.