Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies
This addresses the challenge of adapting language models to specialized domains like biomedical QA with limited resources, though it is incremental as it builds on existing masking techniques.
The paper tackled the problem of limited biomedical QA data by proposing a biomedical entity-aware masking strategy to improve domain adaptation of pre-trained language models, achieving performance on par with state-of-the-art models on several datasets.
Biomedical question-answering (QA) has gained increased attention for its capability to provide users with high-quality information from a vast scientific literature. Although an increasing number of biomedical QA datasets has been recently made available, those resources are still rather limited and expensive to produce. Transfer learning via pre-trained language models (LMs) has been shown as a promising approach to leverage existing general-purpose knowledge. However, finetuning these large models can be costly and time consuming, often yielding limited benefits when adapting to specific themes of specialised domains, such as the COVID-19 literature. To bootstrap further their domain adaptation, we propose a simple yet unexplored approach, which we call biomedical entity-aware masking (BEM). We encourage masked language models to learn entity-centric knowledge based on the pivotal entities characterizing the domain at hand, and employ those entities to drive the LM fine-tuning. The resulting strategy is a downstream process applicable to a wide variety of masked LMs, not requiring additional memory or components in the neural architectures. Experimental results show performance on par with state-of-the-art models on several biomedical QA datasets.