Anatomy of OntoGUM--Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms
This addresses the issue of covert overfitting and lack of generalizability in coreference resolution models for NLP researchers, though it is incremental as it adapts an existing corpus rather than introducing a new method.
The paper tackled the problem of evaluating the generalizability of state-of-the-art coreference resolution algorithms by adapting the GUM corpus to the OntoNotes scheme, creating OntoGUM, and found that out-of-domain evaluation across 12 genres showed nearly 15-20% degradation in performance for both deterministic and deep learning systems.
SOTA coreference resolution produces increasingly impressive scores on the OntoNotes benchmark. However lack of comparable data following the same scheme for more genres makes it difficult to evaluate generalizability to open domain data. Zhu et al. (2021) introduced the creation of the OntoGUM corpus for evaluating geralizability of the latest neural LM-based end-to-end systems. This paper covers details of the mapping process which is a set of deterministic rules applied to the rich syntactic and discourse annotations manually annotated in the GUM corpus. Out-of-domain evaluation across 12 genres shows nearly 15-20% degradation for both deterministic and deep learning systems, indicating a lack of generalizability or covert overfitting in existing coreference resolution models.