Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and Documents
This work provides a domain-specific multilingual embedding model for Switzerland, though it is incremental as it builds on existing SwissBERT with fine-tuning.
The authors fine-tuned the SwissBERT encoder model for embedding sentences and documents in Switzerland's four national languages, achieving improved accuracy over the original SwissBERT and a baseline in document retrieval and text classification tasks.
Encoder models trained for the embedding of sentences or short documents have proven useful for tasks such as semantic search and topic modeling. In this paper, we present a version of the SwissBERT encoder model that we specifically fine-tuned for this purpose. SwissBERT contains language adapters for the four national languages of Switzerland -- German, French, Italian, and Romansh -- and has been pre-trained on a large number of news articles in those languages. Using contrastive learning based on a subset of these articles, we trained a fine-tuned version, which we call SentenceSwissBERT. Multilingual experiments on document retrieval and text classification in a Switzerland-specific setting show that SentenceSwissBERT surpasses the accuracy of the original SwissBERT model and of a comparable baseline. The model is openly available for research use.