Longtonotes: OntoNotes with Longer Coreference Chains
This work addresses the need for longer coreference chains in NLP benchmarks, which is incremental as it extends an existing dataset rather than introducing a new method.
The authors tackled the problem of limited document length in coreference resolution benchmarks by creating LongtoNotes, a manually-curated corpus with documents up to 8 times longer than OntoNotes and 2 times longer than Litbank, and they evaluated state-of-the-art neural models on it to reveal areas for improvement in long-document modeling.
Ontonotes has served as the most important benchmark for coreference resolution. However, for ease of annotation, several long documents in Ontonotes were split into smaller parts. In this work, we build a corpus of coreference-annotated documents of significantly longer length than what is currently available. We do so by providing an accurate, manually-curated, merging of annotations from documents that were split into multiple parts in the original Ontonotes annotation process. The resulting corpus, which we call LongtoNotes contains documents in multiple genres of the English language with varying lengths, the longest of which are up to 8x the length of documents in Ontonotes, and 2x those in Litbank. We evaluate state-of-the-art neural coreference systems on this new corpus, analyze the relationships between model architectures/hyperparameters and document length on performance and efficiency of the models, and demonstrate areas of improvement in long-document coreference modeling revealed by our new corpus. Our data and code is available at: https://github.com/kumar-shridhar/LongtoNotes.