StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
This work addresses the need for better natural language understanding models by proposing an incremental enhancement to BERT that improves accuracy across multiple benchmarks.
The authors tackled the problem of improving language understanding by extending BERT to incorporate language structures through auxiliary pre-training tasks at word and sentence levels, resulting in state-of-the-art performance with scores like 89.0 on GLUE, 93.0 F1 on SQuAD v1.1, and 91.7 accuracy on SNLI.
Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.