Establishing Strong Baselines for the New Decade: Sequence Tagging, Syntactic and Semantic Parsing with BERT
This work provides strong baselines for NLP researchers in parsing and tagging tasks, but it is incremental as it builds on BERT with simplifications rather than introducing new paradigms.
The paper tackled sequence tagging, syntactic parsing, and semantic parsing by applying BERT embeddings to simplify and enhance existing models, achieving an average 2.5% improvement over previous state-of-the-art methods, with up to 7.5% gains in some cases.
This paper presents new state-of-the-art models for three tasks, part-of-speech tagging, syntactic parsing, and semantic parsing, using the cutting-edge contextualized embedding framework known as BERT. For each task, we first replicate and simplify the current state-of-the-art approach to enhance its model efficiency. We then evaluate our simplified approaches on those three tasks using token embeddings generated by BERT. 12 datasets in both English and Chinese are used for our experiments. The BERT models outperform the previously best-performing models by 2.5% on average (7.5% for the most significant case). Moreover, an in-depth analysis on the impact of BERT embeddings is provided using self-attention, which helps understanding in this rich yet representation. All models and source codes are available in public so that researchers can improve upon and utilize them to establish strong baselines for the next decade.