ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning
This work aims to improve event temporal reasoning capabilities for pre-trained language models, which is crucial for event-centric applications.
This paper addresses the struggle of pre-trained language models (PTLMs) with event temporal reasoning by introducing a continual pre-training approach. The proposed framework, ECONET, uses self-supervised learning objectives to enhance PTLMs' attention to event and temporal information, leading to improved fine-tuning performances across five relation extraction and question answering tasks and achieving new or on-par state-of-the-art results in most downstream tasks.
While pre-trained language models (PTLMs) have achieved noticeable success on many NLP tasks, they still struggle for tasks that require event temporal reasoning, which is essential for event-centric applications. We present a continual pre-training approach that equips PTLMs with targeted knowledge about event temporal relations. We design self-supervised learning objectives to recover masked-out event and temporal indicators and to discriminate sentences from their corrupted counterparts (where event or temporal indicators got replaced). By further pre-training a PTLM with these objectives jointly, we reinforce its attention to event and temporal information, yielding enhanced capability on event temporal reasoning. This effective continual pre-training framework for event temporal reasoning (ECONET) improves the PTLMs' fine-tuning performances across five relation extraction and question answering tasks and achieves new or on-par state-of-the-art performances in most of our downstream tasks.