The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding
This addresses the under-explored challenge of enhancing multilingual language models for code-switched NLP tasks, offering a reliable technique with consistent gains across multiple language pairs, though it appears incremental as it builds on existing pretraining methods.
The paper tackled the problem of improving code-switched natural language understanding by proposing bilingual intermediate pretraining, achieving substantial absolute improvements of 7.87%, 20.15%, and 10.99% over previous state-of-the-art systems on Hindi-English NLI, QA, and Spanish-English SA tasks, respectively.
While recent benchmarks have spurred a lot of new work on improving the generalization of pretrained multilingual language models on multilingual tasks, techniques to improve code-switched natural language understanding tasks have been far less explored. In this work, we propose the use of bilingual intermediate pretraining as a reliable technique to derive large and consistent performance gains on three different NLP tasks using code-switched text. We achieve substantial absolute improvements of 7.87%, 20.15%, and 10.99%, on the mean accuracies and F1 scores over previous state-of-the-art systems for Hindi-English Natural Language Inference (NLI), Question Answering (QA) tasks, and Spanish-English Sentiment Analysis (SA) respectively. We show consistent performance gains on four different code-switched language-pairs (Hindi-English, Spanish-English, Tamil-English and Malayalam-English) for SA. We also present a code-switched masked language modelling (MLM) pretraining technique that consistently benefits SA compared to standard MLM pretraining using real code-switched text.