An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks
This work addresses the need for more sophisticated structured prediction in NLP, offering incremental improvements for tasks like semantic role labeling and parsing.
The paper tackled the problem of capturing complex dependencies in sequence labeling tasks by proposing high-order energy terms with neural parameterizations, achieving substantial improvements on four tasks while maintaining decoding speed comparable to local classifiers.
Many tasks in natural language processing involve predicting structured outputs, e.g., sequence labeling, semantic role labeling, parsing, and machine translation. Researchers are increasingly applying deep representation learning to these problems, but the structured component of these approaches is usually quite simplistic. In this work, we propose several high-order energy terms to capture complex dependencies among labels in sequence labeling, including several that consider the entire label sequence. We use neural parameterizations for these energy terms, drawing from convolutional, recurrent, and self-attention networks. We use the framework of learning energy-based inference networks (Tu and Gimpel, 2018) for dealing with the difficulties of training and inference with such models. We empirically demonstrate that this approach achieves substantial improvement using a variety of high-order energy terms on four sequence labeling tasks, while having the same decoding speed as simple, local classifiers. We also find high-order energies to help in noisy data conditions.