Adapting Monolingual Models: Data can be Scarce when Language Similarity is High
This addresses the challenge of resource scarcity for minority languages, offering a practical solution for NLP applications in such contexts, though it is incremental as it builds on existing BERT-based methods.
The paper tackles the problem of training large models for low-resource languages by investigating zero-shot transfer learning with minimal data, showing that retraining lexical layers of BERT-based models with as little as 10MB of data can achieve high performance in POS-tagging tasks when language similarity is high.
For many (minority) languages, the resources needed to train large models are not available. We investigate the performance of zero-shot transfer learning with as little data as possible, and the influence of language similarity in this process. We retrain the lexical layers of four BERT-based models using data from two low-resource target language varieties, while the Transformer layers are independently fine-tuned on a POS-tagging task in the model's source language. By combining the new lexical layers and fine-tuned Transformer layers, we achieve high task performance for both target languages. With high language similarity, 10MB of data appears sufficient to achieve substantial monolingual transfer performance. Monolingual BERT-based models generally achieve higher downstream task performance after retraining the lexical layer than multilingual BERT, even when the target language is included in the multilingual model.