Assessment of Amazon Comprehend Medical: Medication Information Extraction
This independent validation assesses a commercial clinical NLP tool for healthcare applications, showing it is competitive but incremental compared to existing research systems.
The researchers evaluated Amazon Comprehend Medical's medication extraction capability using benchmark datasets from the 2009 i2b2 and 2018 n2c2 challenges, where it achieved F-scores of 0.768 and 0.828, ranking lowest compared to top systems, and an F-score of 0.753 on internal clinical notes.
In November 27, 2018, Amazon Web Services (AWS) released Amazon Comprehend Medical (ACM), a deep learning based system that automatically extracts clinical concepts (which include anatomy, medical conditions, protected health information (PH)I, test names, treatment names, and medical procedures, and medications) from clinical text notes. Uptake and trust in any new data product relies on independent validation across benchmark datasets and tools to establish and confirm expected quality of results. This work focuses on the medication extraction task, and particularly, ACM was evaluated using the official test sets from the 2009 i2b2 Medication Extraction Challenge and 2018 n2c2 Track 2: Adverse Drug Events and Medication Extraction in EHRs. Overall, ACM achieved F-scores of 0.768 and 0.828. These scores ranked the lowest when compared to the three best systems in the respective challenges. To further establish the generalizability of its medication extraction performance, a set of random internal clinical text notes from NYU Langone Medical Center were also included in this work. And in this corpus, ACM garnered an F-score of 0.753.