Extensive Evaluation of Transformer-based Architectures for Adverse Drug Events Extraction
This work addresses the problem of selecting effective models for adverse event extraction in digital pharmacovigilance, particularly from informal sources like forums and tweets, but it is incremental as it focuses on comparative analysis rather than introducing new methods.
The paper conducted an extensive evaluation of 19 Transformer-based architectures for adverse drug events extraction from informal texts, comparing their performance on two datasets and analyzing the impact of additional processing layers and model features, identifying key insights from the experimental data.
Adverse Event (ADE) extraction is one of the core tasks in digital pharmacovigilance, especially when applied to informal texts. This task has been addressed by the Natural Language Processing community using large pre-trained language models, such as BERT. Despite the great number of Transformer-based architectures used in the literature, it is unclear which of them has better performances and why. Therefore, in this paper we perform an extensive evaluation and analysis of 19 Transformer-based models for ADE extraction on informal texts. We compare the performance of all the considered models on two datasets with increasing levels of informality (forums posts and tweets). We also combine the purely Transformer-based models with two commonly-used additional processing layers (CRF and LSTM), and analyze their effect on the models performance. Furthermore, we use a well-established feature importance technique (SHAP) to correlate the performance of the models with a set of features that describe them: model category (AutoEncoding, AutoRegressive, Text-to-Text), pretraining domain, training from scratch, and model size in number of parameters. At the end of our analyses, we identify a list of take-home messages that can be derived from the experimental data.