Transition-based Parsing with Stack-Transformers
This work addresses parsing efficiency and accuracy for NLP researchers, but it is incremental as it builds on existing Transformer architectures.
The paper tackled the problem of modeling parser state in transition-based parsing by modifying the cross-attention mechanism in Transformers to handle global or local states, resulting in improved performance on dependency and AMR parsing tasks, especially with smaller models or limited data.
Modeling the parser state is key to good performance in transition-based parsing. Recurrent Neural Networks considerably improved the performance of transition-based systems by modelling the global state, e.g. stack-LSTM parsers, or local state modeling of contextualized features, e.g. Bi-LSTM parsers. Given the success of Transformer architectures in recent parsing systems, this work explores modifications of the sequence-to-sequence Transformer architecture to model either global or local parser states in transition-based parsing. We show that modifications of the cross attention mechanism of the Transformer considerably strengthen performance both on dependency and Abstract Meaning Representation (AMR) parsing tasks, particularly for smaller models or limited training data.