Modular Adaptation of Multilingual Encoders to Written Swiss German Dialect
This addresses the problem of limited training data and dialectal variation for Swiss German NLP applications, representing an incremental improvement.
The paper tackled the challenge of creating neural text encoders for written Swiss German by adapting multilingual encoders, finding that adding a Swiss German adapter achieves 97.5% of the performance of full adaptation and that character-level models are more effective for retrieval tasks.
Creating neural text encoders for written Swiss German is challenging due to a dearth of training data combined with dialectal variation. In this paper, we build on several existing multilingual encoders and adapt them to Swiss German using continued pre-training. Evaluation on three diverse downstream tasks shows that simply adding a Swiss German adapter to a modular encoder achieves 97.5% of fully monolithic adaptation performance. We further find that for the task of retrieving Swiss German sentences given Standard German queries, adapting a character-level model is more effective than the other adaptation strategies. We release our code and the models trained for our experiments at https://github.com/ZurichNLP/swiss-german-text-encoders