Relation Extraction in underexplored biomedical domains: A diversity-optimised sampling and synthetic data generation approach
This work addresses the data sparsity issue for researchers and practitioners in biomedical domains like drug-discovery, focusing on natural-products literature, and is incremental as it builds on existing methods with new sampling and data generation techniques.
The paper tackled the problem of sparse labeled data for Relation Extraction in biomedical domains, specifically natural-products literature, by creating the first curated evaluation dataset and using a diversity-optimised sampler and synthetic data generation; the result was a BioGPT-Large model achieving an f1-score of 59.0 for end-to-end RE, with models showing substantial improvements when fine-tuned on synthetic data.
The sparsity of labelled data is an obstacle to the development of Relation Extraction models and the completion of databases in various biomedical areas. While being of high interest in drug-discovery, the natural-products literature, reporting the identification of potential bioactive compounds from organisms, is a concrete example of such an overlooked topic. To mark the start of this new task, we created the first curated evaluation dataset and extracted literature items from the LOTUS database to build training sets. To this end, we developed a new sampler inspired by diversity metrics in ecology, named Greedy Maximum Entropy sampler, or GME-sampler (https://github.com/idiap/gme-sampler). The strategic optimization of both balance and diversity of the selected items in the evaluation set is important given the resource-intensive nature of manual curation. After quantifying the noise in the training set, in the form of discrepancies between the input abstracts text and the expected output labels, we explored different strategies accordingly. Framing the task as an end-to-end Relation Extraction, we evaluated the performance of standard fine-tuning as a generative task and few-shot learning with open Large Language Models (LLaMA 7B-65B). In addition to their evaluation in few-shot settings, we explore the potential of open Large Language Models (Vicuna-13B) as synthetic data generator and propose a new workflow for this purpose. All evaluated models exhibited substantial improvements when fine-tuned on synthetic abstracts rather than the original noisy data. We provide our best performing (f1-score=59.0) BioGPT-Large model for end-to-end RE of natural-products relationships along with all the generated synthetic data and the evaluation dataset. See more details at https://github.com/idiap/abroad-re.