Are LLMs Good Annotators for Discourse-level Event Relation Extraction?
This work addresses the effectiveness of LLMs for discourse-level event relation extraction, an incremental study highlighting limitations in a specific NLP domain.
The paper tackled the problem of using Large Language Models (LLMs) for discourse-level event relation extraction, finding that LLMs notably underperform compared to supervised baselines, with GPT-3.5 and LLaMA-2 showing weaknesses such as fabricating event mentions and failing to capture long-distance relations.
Large Language Models (LLMs) have demonstrated proficiency in a wide array of natural language processing tasks. However, its effectiveness over discourse-level event relation extraction (ERE) tasks remains unexplored. In this paper, we assess the effectiveness of LLMs in addressing discourse-level ERE tasks characterized by lengthy documents and intricate relations encompassing coreference, temporal, causal, and subevent types. Evaluation is conducted using an commercial model, GPT-3.5, and an open-source model, LLaMA-2. Our study reveals a notable underperformance of LLMs compared to the baseline established through supervised learning. Although Supervised Fine-Tuning (SFT) can improve LLMs performance, it does not scale well compared to the smaller supervised baseline model. Our quantitative and qualitative analysis shows that LLMs have several weaknesses when applied for extracting event relations, including a tendency to fabricate event mentions, and failures to capture transitivity rules among relations, detect long distance relations, or comprehend contexts with dense event mentions. Code available at: https://github.com/WeiKangda/LLM-ERE.git.