A Multi-Axis Annotation Scheme for Event Temporal Relations
This work addresses the issue of inconsistent annotations for event temporal relations, which is incremental as it refines existing schemes to enable more reliable data collection.
The paper tackled the problem of low inter-annotator agreement in temporal relation annotation by proposing a multi-axis modeling scheme and focusing on start-points only, resulting in a significant improvement in Cohen's Kappa from the 60s to the 80s.
Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation using the proposed scheme shows significant improvement in IAA from the conventional 60's to 80's (Cohen's Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.