Stress Test Evaluation of Biomedical Word Embeddings
This work addresses the lack of research on quantifying the behavior of biomedical NLP models under severe stress scenarios, which is important for developers and users in the biomedical domain, though it is incremental as it applies existing adversarial methods to this specific area.
The paper tackled the problem of evaluating the robustness of biomedical word embeddings under adversarial stress scenarios, such as spelling errors and synonyms, and found that performance decreased considerably but could be improved with adversarial training, sometimes exceeding original performance.
The success of pretrained word embeddings has motivated their use in the biomedical domain, with contextualized embeddings yielding remarkable results in several biomedical NLP tasks. However, there is a lack of research on quantifying their behavior under severe "stress" scenarios. In this work, we systematically evaluate three language models with adversarial examples -- automatically constructed tests that allow us to examine how robust the models are. We propose two types of stress scenarios focused on the biomedical named entity recognition (NER) task, one inspired by spelling errors and another based on the use of synonyms for medical terms. Our experiments with three benchmarks show that the performance of the original models decreases considerably, in addition to revealing their weaknesses and strengths. Finally, we show that adversarial training causes the models to improve their robustness and even to exceed the original performance in some cases.