ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes
This addresses the need for better NLP systems in healthcare to process patient histories over time, though it is incremental as it adapts an existing model to a specific domain.
The paper tackled the problem of interpreting longitudinal clinical notes by introducing ClinicalMamba, a specialized Mamba model pretrained on a large corpus, which outperformed existing models like Mamba, clinical Llama, and GPT-4 in speed and accuracy for information extraction tasks.
The advancement of natural language processing (NLP) systems in healthcare hinges on language model ability to interpret the intricate information contained within clinical notes. This process often requires integrating information from various time points in a patient's medical history. However, most earlier clinical language models were pretrained with a context length limited to roughly one clinical document. In this study, We introduce ClinicalMamba, a specialized version of the Mamba language model, pretrained on a vast corpus of longitudinal clinical notes to address the unique linguistic characteristics and information processing needs of the medical domain. ClinicalMamba, with 130 million and 2.8 billion parameters, demonstrates a superior performance in modeling clinical language across extended text lengths compared to Mamba and clinical Llama. With few-shot learning, ClinicalMamba achieves notable benchmarks in speed and accuracy, outperforming existing clinical language models and general domain large models like GPT-4 in longitudinal clinical notes information extraction tasks.