Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study
This work addresses the challenge of handling long sequences in French biomedical NLP, which is incremental as it adapts existing models rather than introducing new paradigms.
The study tackled the problem of applying French biomedical pretrained models to long clinical documents by comparing three adaptation strategies using the Longformer architecture, finding that further pre-training an English clinical model with French texts outperformed other methods on 16 downstream tasks, with long-sequence models improving performance across most tasks but BERT models remaining most efficient for named entity recognition.
Recently, pretrained language models based on BERT have been introduced for the French biomedical domain. Although these models have achieved state-of-the-art results on biomedical and clinical NLP tasks, they are constrained by a limited input sequence length of 512 tokens, which poses challenges when applied to clinical notes. In this paper, we present a comparative study of three adaptation strategies for long-sequence models, leveraging the Longformer architecture. We conducted evaluations of these models on 16 downstream tasks spanning both biomedical and clinical domains. Our findings reveal that further pre-training an English clinical model with French biomedical texts can outperform both converting a French biomedical BERT to the Longformer architecture and pre-training a French biomedical Longformer from scratch. The results underscore that long-sequence French biomedical models improve performance across most downstream tasks regardless of sequence length, but BERT based models remain the most efficient for named entity recognition tasks.