An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain
This work addresses the practical cost-benefit trade-off of using large pre-trained models for biomedical NLP researchers and practitioners, suggesting that domain-specific pre-training might be an incremental gain for some.
This paper evaluates the effectiveness of transformer-based language models in the biomedical domain, specifically replicating BioBERT and investigating domain-specific versus domain-agnostic pre-trained models. It found that while pre-trained models can be impactful for tasks like Question Answering and Named Entity Recognition, the improvement may not always justify the high cost of domain-specific pre-training.
With the growing amount of text in health data, there have been rapid advances in large pre-trained models that can be applied to a wide variety of biomedical tasks with minimal task-specific modifications. Emphasizing the cost of these models, which renders technical replication challenging, this paper summarizes experiments conducted in replicating BioBERT and further pre-training and careful fine-tuning in the biomedical domain. We also investigate the effectiveness of domain-specific and domain-agnostic pre-trained models across downstream biomedical NLP tasks. Our finding confirms that pre-trained models can be impactful in some downstream NLP tasks (QA and NER) in the biomedical domain; however, this improvement may not justify the high cost of domain-specific pre-training.