Validation and Transparency in AI systems for pharmacovigilance: a case study applied to the medical literature monitoring of adverse events
This work addresses the problem of high screening burden in pharmacovigilance for regulatory and industry users, but it is incremental as it applies existing guidance and methods to a specific domain task.
The paper tackles the challenge of reducing effort in pharmacovigilance by developing an AI system for medical literature monitoring of adverse events, achieving a 55% reduction in screening irrelevant articles while maintaining 0.99 recall on suspected adverse articles.
Recent advances in artificial intelligence applied to biomedical text are opening exciting opportunities for improving pharmacovigilance activities currently burdened by the ever growing volumes of real world data. To fully realize these opportunities, existing regulatory guidance and industry best practices should be taken into consideration in order to increase the overall trustworthiness of the system and enable broader adoption. In this paper we present a case study on how to operationalize existing guidance for validated AI systems in pharmacovigilance focusing on the specific task of medical literature monitoring (MLM) of adverse events from the scientific literature. We describe an AI system designed with the goal of reducing effort in MLM activities built in close collaboration with subject matter experts and considering guidance for validated systems in pharmacovigilance and AI transparency. In particular we make use of public disclosures as a useful risk control measure to mitigate system misuse and earn user trust. In addition we present experimental results showing the system can significantly remove screening effort while maintaining high levels of recall (filtering 55% of irrelevant articles on average, for a target recall of 0.99 on suspected adverse articles) and provide a robust method for tuning the desired recall to suit a particular risk profile.