PIXAR: Auto-Regressive Language Modeling in Pixel Space
This work addresses the challenge of building open-vocabulary language models that operate solely on perceptual inputs, potentially questioning the need for symbolic text representations, though it is incremental in extending pixel-based models to generation.
The authors tackled the problem of pixel-based language models being limited to discriminative tasks by introducing PIXAR, the first autoregressive model that generates text directly from pixel representations, achieving improvements of 8.1 on LAMBADA and 8.5 on bAbI to match GPT-2 performance.
Recent work showed the possibility of building open-vocabulary large language models (LLMs) that directly operate on pixel representations. These models are implemented as autoencoders that reconstruct masked patches of rendered text. However, these pixel-based LLMs are limited to discriminative tasks (e.g., classification) and, similar to BERT, cannot be used to generate text. Therefore, they cannot be used for generative tasks such as free-form question answering. In this work, we introduce PIXAR, the first pixel-based autoregressive LLM that performs text generation. Consisting of only a decoder, PIXAR can perform free-form generative tasks while keeping the number of parameters on par with previous encoder-decoder models. Furthermore, we highlight the challenges of generating text as non-noisy images and show this is due to using a maximum likelihood objective. To overcome this problem, we propose an adversarial pretraining stage that improves the readability and accuracy of PIXAR by 8.1 on LAMBADA and 8.5 on bAbI -- making it comparable to GPT-2 on text generation tasks. This paves the way to build open-vocabulary LLMs that operate on perceptual input only and calls into question the necessity of the usual symbolic input representation, i.e., text as (sub)tokens.