Multilingual Pretraining for Pixel Language Models
This work addresses the need for more effective multilingual support in pixel language models, which is incremental as it builds on existing models by extending pretraining to multiple languages.
The paper tackled the problem of underexplored multilingual pretraining for pixel language models by introducing PIXEL-M4, pretrained on four diverse languages, and showed it outperforms an English-only counterpart on non-Latin scripts in semantic and syntactic tasks.
Pixel language models operate directly on images of rendered text, eliminating the need for a fixed vocabulary. While these models have demonstrated strong capabilities for downstream cross-lingual transfer, multilingual pretraining remains underexplored. We introduce PIXEL-M4, a model pretrained on four visually and linguistically diverse languages: English, Hindi, Ukrainian, and Simplified Chinese. Multilingual evaluations on semantic and syntactic tasks show that PIXEL-M4 outperforms an English-only counterpart on non-Latin scripts. Word-level probing analyses confirm that PIXEL-M4 captures rich linguistic features, even in languages not seen during pretraining. Furthermore, an analysis of its hidden representations shows that multilingual pretraining yields a semantic embedding space closely aligned across the languages used for pretraining. This work demonstrates that multilingual pretraining substantially enhances the capability of pixel language models to effectively support a diverse set of languages.