Autoregressive Structured Prediction with Language Models
This addresses a key bottleneck in NLP for researchers and practitioners by improving structured prediction performance with language models, though it is incremental as it builds on existing autoregressive methods.
The paper tackles the problem of representing structured outputs like named entities and coreference chains for pretrained language models, which often leads to inferior performance due to flattening sequences. It introduces an autoregressive approach that models structures as action sequences, achieving new state-of-the-art results on tasks such as named entity recognition, relation extraction, and coreference resolution.
Recent years have seen a paradigm shift in NLP towards using pretrained language models ({PLM}) for a wide range of tasks. However, there are many difficult design decisions to represent structures (e.g. tagged text, coreference chains) in a way such that they can be captured by PLMs. Prior work on structured prediction with PLMs typically flattens the structured output into a sequence, which limits the quality of structural information being learned and leads to inferior performance compared to classic discriminative models. In this work, we describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs, allowing in-structure dependencies to be learned without any loss. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at, namely, named entity recognition, end-to-end relation extraction, and coreference resolution.