Improving Open Information Extraction with Large Language Models: A Study on Demonstration Uncertainty
This work addresses the challenge of improving OIE for natural language processing applications, but it is incremental as it builds on existing in-context learning and uncertainty quantification techniques.
The paper tackled the problem of large language models (LLMs) underperforming in Open Information Extraction (OIE) tasks by addressing issues with irrelevant context and low confidence in generated relations, resulting in an approach that competes with supervised methods on three benchmark datasets.
Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text, typically in the form of (subject, relation, object) triples. Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks due to two key issues. First, LLMs struggle to distinguish irrelevant context from relevant relations and generate structured output due to the restrictions on fine-tuning the model. Second, LLMs generates responses autoregressively based on probability, which makes the predicted relations lack confidence. In this paper, we assess the capabilities of LLMs in improving the OIE task. Particularly, we propose various in-context learning strategies to enhance LLM's instruction-following ability and a demonstration uncertainty quantification module to enhance the confidence of the generated relations. Our experiments on three OIE benchmark datasets show that our approach holds its own against established supervised methods, both quantitatively and qualitatively.