MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts
This work advances multilingual pixel-based language modeling, offering a tokenization-free alternative that generalizes across scripts and languages, which is important for NLP applications in low-resource or diverse linguistic settings.
MIXAR is the first generative pixel-based language model trained on eight languages with diverse scripts, achieving substantial performance improvements over previous pixel-based and tokenizer-based models on multilingual tasks, and showing robustness to unseen languages and input perturbations, with further gains at 0.5B parameters.
Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges. However, the inherent perceptual diversity across languages poses a significant hurdle for multilingual generalization in pixel space. This paper introduces MIXAR, the first generative pixel-based language model trained on eight different languages utilizing a range of different scripts. We empirically evaluate MIXAR against previous pixel-based models as well as comparable tokenizer-based models, demonstrating substantial performance improvement on discriminative and generative multilingual tasks. Additionally, we show how MIXAR is robust to languages never seen during the training. These results are further strengthened when scaling the model to 0.5B parameters which not only improves its capabilities in generative tasks like LAMBADA but also its robustness when challenged with input perturbations such as orthographic attacks.