Syntax-augmented Multilingual BERT for Cross-lingual Transfer
This work addresses the problem of typological differences hindering cross-lingual transfer for NLP practitioners, offering an incremental enhancement by augmenting existing models with syntax.
The paper tackles the challenge of cross-lingual transfer in multilingual text encoders by explicitly incorporating language syntax, such as syntactic dependencies, into mBERT training, resulting in average improvements of 1.4 and 1.6 points on PAWS-X and MLQA benchmarks, with significant boosts of 3.9 and 3.1 points in generalized transfer settings.
In recent years, we have seen a colossal effort in pre-training multilingual text encoders using large-scale corpora in many languages to facilitate cross-lingual transfer learning. However, due to typological differences across languages, the cross-lingual transfer is challenging. Nevertheless, language syntax, e.g., syntactic dependencies, can bridge the typological gap. Previous works have shown that pre-trained multilingual encoders, such as mBERT \cite{devlin-etal-2019-bert}, capture language syntax, helping cross-lingual transfer. This work shows that explicitly providing language syntax and training mBERT using an auxiliary objective to encode the universal dependency tree structure helps cross-lingual transfer. We perform rigorous experiments on four NLP tasks, including text classification, question answering, named entity recognition, and task-oriented semantic parsing. The experiment results show that syntax-augmented mBERT improves cross-lingual transfer on popular benchmarks, such as PAWS-X and MLQA, by 1.4 and 1.6 points on average across all languages. In the \emph{generalized} transfer setting, the performance boosted significantly, with 3.9 and 3.1 points on average in PAWS-X and MLQA.