ELECTRAMed: a new pre-trained language representation model for biomedical NLP
This work addresses the problem of processing large volumes of biomedical texts for researchers and practitioners, though it is incremental as it adapts an existing efficient architecture to a specialized domain.
The authors tackled the need for effective biomedical NLP models by proposing ELECTRAMed, a domain-specific pre-trained language model based on the ELECTRA architecture, which achieved state-of-the-art results on the BC5CDR corpus for named entity recognition and performed best in 2 out of 5 runs on the BioASQ-factoid Challenge for question answering.
The overwhelming amount of biomedical scientific texts calls for the development of effective language models able to tackle a wide range of biomedical natural language processing (NLP) tasks. The most recent dominant approaches are domain-specific models, initialized with general-domain textual data and then trained on a variety of scientific corpora. However, it has been observed that for specialized domains in which large corpora exist, training a model from scratch with just in-domain knowledge may yield better results. Moreover, the increasing focus on the compute costs for pre-training recently led to the design of more efficient architectures, such as ELECTRA. In this paper, we propose a pre-trained domain-specific language model, called ELECTRAMed, suited for the biomedical field. The novel approach inherits the learning framework of the general-domain ELECTRA architecture, as well as its computational advantages. Experiments performed on benchmark datasets for several biomedical NLP tasks support the usefulness of ELECTRAMed, which sets the novel state-of-the-art result on the BC5CDR corpus for named entity recognition, and provides the best outcome in 2 over the 5 runs of the 7th BioASQ-factoid Challange for the question answering task.