Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach
This addresses robustness issues in language models for users in multilingual or noisy-text environments, representing a novel approach rather than an incremental improvement.
The paper tackles the vulnerability of autoregressive language models to orthographic attacks by proposing a pixel-based generative language model that renders words as images, which demonstrates resilience to orthographic noise and effectiveness in multilingual settings on datasets like LAMBADA, WMT24, and SST-2.
Autoregressive language models are vulnerable to orthographic attacks, where input text is perturbed with characters from multilingual alphabets, leading to substantial performance degradation. This vulnerability primarily stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings. To address this limitation, we propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images. This design provides stronger robustness to noisy inputs, while an extension of compatibility to multilingual text across diverse writing systems. We evaluate the proposed method on the multilingual LAMBADA dataset, WMT24 dataset and the SST-2 benchmark, demonstrating both its resilience to orthographic noise and its effectiveness in multilingual settings.