Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing
This addresses the problem of dependency parsing across languages without parallel data, offering an efficient solution for NLP researchers, though it is incremental as it builds on existing BERT models.
The paper tackles zero-shot cross-lingual dependency parsing by proposing Cross-Lingual BERT Transformation (CLBT), a linear method to align contextual embeddings from pre-trained BERT models, resulting in substantial performance gains over static embeddings and competitive results compared to XLM.
This paper investigates the problem of learning cross-lingual representations in a contextual space. We propose Cross-Lingual BERT Transformation (CLBT), a simple and efficient approach to generate cross-lingual contextualized word embeddings based on publicly available pre-trained BERT models (Devlin et al., 2018). In this approach, a linear transformation is learned from contextual word alignments to align the contextualized embeddings independently trained in different languages. We demonstrate the effectiveness of this approach on zero-shot cross-lingual transfer parsing. Experiments show that our embeddings substantially outperform the previous state-of-the-art that uses static embeddings. We further compare our approach with XLM (Lample and Conneau, 2019), a recently proposed cross-lingual language model trained with massive parallel data, and achieve highly competitive results.