An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels
This finding questions fundamental design assumptions in computer vision architectures, though it is computationally impractical and incremental in method.
The paper demonstrates that vanilla Transformers can achieve high performance by treating individual pixels as tokens, challenging the necessity of locality inductive biases in vision architectures, with results shown across classification, regression, self-supervised learning, and image generation tasks.
This work does not introduce a new method. Instead, we present an interesting finding that questions the necessity of the inductive bias of locality in modern computer vision architectures. Concretely, we find that vanilla Transformers can operate by directly treating each individual pixel as a token and achieve highly performant results. This is substantially different from the popular design in Vision Transformer, which maintains the inductive bias from ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a token). We showcase the effectiveness of pixels-as-tokens across three well-studied computer vision tasks: supervised learning for classification and regression, self-supervised learning via masked autoencoding, and image generation with diffusion models. Although it's computationally less practical to directly operate on individual pixels, we believe the community must be made aware of this surprising piece of knowledge when devising the next generation of neural network architectures for computer vision.