Leveraging ChatGPT in Pharmacovigilance Event Extraction: An Empirical Study
This addresses the problem of automating pharmacovigilance event extraction for medical researchers, but it is incremental as it shows limitations of LLMs in this domain.
This study investigated ChatGPT's ability to extract adverse events from medical texts for pharmacovigilance, finding it performs reasonably with good prompts but underperforms fine-tuned small models, and data augmentation with its outputs can degrade performance unless filtered.
With the advent of large language models (LLMs), there has been growing interest in exploring their potential for medical applications. This research aims to investigate the ability of LLMs, specifically ChatGPT, in the context of pharmacovigilance event extraction, of which the main goal is to identify and extract adverse events or potential therapeutic events from textual medical sources. We conduct extensive experiments to assess the performance of ChatGPT in the pharmacovigilance event extraction task, employing various prompts and demonstration selection strategies. The findings demonstrate that while ChatGPT demonstrates reasonable performance with appropriate demonstration selection strategies, it still falls short compared to fully fine-tuned small models. Additionally, we explore the potential of leveraging ChatGPT for data augmentation. However, our investigation reveals that the inclusion of synthesized data into fine-tuning may lead to a decrease in performance, possibly attributed to noise in the ChatGPT-generated labels. To mitigate this, we explore different filtering strategies and find that, with the proper approach, more stable performance can be achieved, although constant improvement remains elusive.