Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
This work addresses the prohibitive resource requirements for pretraining large NLP models, making it more affordable, though it is an incremental improvement on existing Transformer architectures.
The authors tackled the high computational cost of pretraining Transformer models like BERT by introducing Shatter, an alternative self-attention architecture that uses relative sequence partitioning and single-headed attention, achieving better performance than BERT with 15% faster pretraining per step, over 50% memory savings, and matching BERT_Base performance in 7 days on 8 GPUs.
The highly popular Transformer architecture, based on self-attention, is the foundation of large pretrained models such as BERT, that have become an enduring paradigm in NLP. While powerful, the computational resources and time required to pretrain such models can be prohibitive. In this work, we present an alternative self-attention architecture, Shatter, that more efficiently encodes sequence information by softly partitioning the space of relative positions and applying different value matrices to different parts of the sequence. This mechanism further allows us to simplify the multi-headed attention in Transformer to single-headed. We conduct extensive experiments showing that Shatter achieves better performance than BERT, with pretraining being faster per step (15% on TPU), converging in fewer steps, and offering considerable memory savings (>50%). Put together, Shatter can be pretrained on 8 V100 GPUs in 7 days, and match the performance of BERT_Base -- making the cost of pretraining much more affordable.