Towards Understanding Self-Pretraining for Sequence Classification
For researchers studying Transformer optimization and pretraining, this work provides mechanistic understanding of why masked pretraining helps in sequence classification, though it is an incremental analysis of existing findings.
This paper investigates why self-pretraining (SPT) improves Transformer performance in sequence classification, finding that SPT helps learn useful query-key attention patterns (specifically proximity interactions) that label supervision alone struggles to learn from random initialization. Ablations show SPT's gains stem from learning proximity-biased attention, not depth or generalization issues.
Amos et al. (2024) showed that the accuracy of Transformer models in sequence classification can be significantly improved by first pretraining with a masked token prediction objective without external data or augmentation, a procedure referred to as self-pretraining (SPT). While the primary objective of Amos et al. (2024) was to showcase that Transformers can achieve strong performance on the Long-Range Arena (LRA), their pipeline raises more fundamental questions: How does SPT drive optimization to better solutions? Why can standard supervised training fail in Transformers? To better understand this, we replicate and systematically ablate the findings of Amos et al. (2024). Our ablations suggest that a central bottleneck in the studied settings is not depth or generalization alone, but the ability of label supervision to learn useful query-key Attention patterns from random initialization. With a minimal setup, we identify learning proximity interactions - turning absolute positional encodings into proximity-biased Attention scores - as a key source of the improvements brought by SPT. Finally, in a simplified theoretical setup, we show that label supervision can be locally blind to certain Attention-score directions that are instead detectable through masked reconstruction.