Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
This work addresses data efficiency and domain-agnostic pretraining for vision transformers, offering an incremental but promising strategy for machine learning practitioners.
The paper tackles the problem of improving vision transformers' data efficiency and convergence by pretraining them on procedurally-generated data without visual content, resulting in a 1.7% accuracy improvement on ImageNet-1k with only 1% of the training budget allocated to this data.
Transformers show remarkable versatility across domains, suggesting the existence of inductive biases beneficial across modalities. In this work, we explore a new way to instil such generic biases in vision transformers (ViTs) by pretraining on procedurally-generated data devoid of visual or semantic content. We generate this data with simple algorithms such as formal grammars, so the results bear no relationship to either natural or synthetic images. We use this procedurally-generated data to pretrain ViTs in a warm-up phase that bypasses their visual patch embedding mechanisms, thus encouraging the models to internalise abstract computational priors. When followed by standard image-based training, this warm-up significantly improves data efficiency, convergence speed, and downstream performance. On ImageNet-1k for example, allocating just 1% of the training budget to procedural data improves final accuracy by over 1.7%. In terms of its effect on performance, 1% procedurally generated data is thus equivalent to 28% of the ImageNet-1k data. These findings suggest a promising path toward new data-efficient and domain-agnostic pretraining strategies.