MrBERT: Modern Multilingual Encoders via Vocabulary, Domain, and Dimensional Adaptation
This work addresses the need for efficient, domain-specialized multilingual models for production use in high-stakes applications.
The paper introduces MrBERT, a family of multilingual encoders that achieves state-of-the-art results on Catalan- and Spanish-specific tasks while maintaining robust performance in biomedical and legal domains, and incorporates Matryoshka Representation Learning to reduce inference and storage costs.
We introduce MrBERT, a family of 150M-300M parameter encoders built on the ModernBERT architecture and pre-trained on 35 languages and code. Through targeted adaptation, this model family achieves state-of-the-art results on Catalan- and Spanish-specific tasks, while establishing robust performance across specialized biomedical and legal domains. To bridge the gap between research and production, we incorporate Matryoshka Representation Learning (MRL), enabling flexible vector sizing that significantly reduces inference and storage costs. Ultimately, the MrBERT family demonstrates that modern encoder architectures can be optimized for both localized linguistic excellence and efficient, high-stakes domain specialization. We open source the complete model family on Huggingface.