Subjectivity in the Annotation of Bridging Anaphora
This addresses annotation inconsistency in NLP for bridging anaphora, which is incremental as it builds on existing schemes.
The study tackled the problem of inconsistent annotation in bridging anaphora by analyzing subjectivity at three levels—anaphor recognition, antecedent resolution, and subtype selection—using a pilot annotation on the GUM corpus. Results indicated that previous resources are likely under-annotated, with low annotator overlap for exhaustive identification and moderate agreement on subtypes.
Bridging refers to the associative relationship between inferable entities in a discourse and the antecedents which allow us to understand them, such as understanding what "the door" means with respect to an aforementioned "house". As identifying associative relations between entities is an inherently subjective task, it is difficult to achieve consistent agreement in the annotation of bridging anaphora and their antecedents. In this paper, we explore the subjectivity involved in the annotation of bridging instances at three levels: anaphor recognition, antecedent resolution, and bridging subtype selection. To do this, we conduct an annotation pilot on the test set of the existing GUM corpus, and propose a newly developed classification system for bridging subtypes, which we compare to previously proposed schemes. Our results suggest that some previous resources are likely to be severely under-annotated. We also find that while agreement on the bridging subtype category was moderate, annotator overlap for exhaustively identifying instances of bridging is low, and that many disagreements resulted from subjective understanding of the entities involved.