Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
This work addresses the problem of understanding and improving the data efficiency and robustness of language models for researchers in AI and machine learning, though it is incremental in exploring synthetic data benefits.
The paper found that pretraining small transformers on synthetic procedural data instills distinct, modular algorithmic reasoning structures in different model components, with attention layers often being most transferable and these structures being composable to reinforce multiple capabilities.
Pretraining on large, semantically rich datasets is key for developing language models. Surprisingly, recent studies have shown that even synthetic data, generated procedurally through simple semantic-free algorithms, can yield some of the same benefits as natural language pretraining. It is unclear what specific capabilities such simple synthetic data instils in a model, where these capabilities reside in the architecture, and how they manifest within its weights. In this short paper, we identify several beneficial forms of procedural data, together with specific algorithmic reasoning skills that improve in small transformers. Our core finding is that different procedural rules instil distinct but complementary inductive structures in the model. With extensive ablations and partial-transfer experiments, we discover that these structures reside in different parts of the model. Attention layers often carry the most transferable information, but some pretraining rules impart useful structure to MLP blocks instead. Most interestingly, the structures induced by multiple rules can be composed to jointly reinforce multiple capabilities. These results suggest an exciting possibility of disentangling the acquisition of knowledge from reasoning in language models, with the goal of improving their robustness and data efficiency.