A Study on Transformer Configuration and Training Objective
This work addresses the over-smoothing issue in deep transformer training for vision and language tasks, offering an incremental improvement over existing methods.
The paper tackles the problem of conventional transformer configurations by proposing deeper and narrower architectures for masked autoencoder training, achieving 87.1% top-1 accuracy on ImageNet and outperforming BERT by 1.1 points on GLUE datasets.
Transformer-based models have delivered impressive results on many tasks, particularly vision and language tasks. In many model training situations, conventional configurations are typically adopted. For example, we often set the base model with hidden dimensions (i.e. model width) to be 768 and the number of transformer layers (i.e. model depth) to be 12. In this paper, we revisit these conventional configurations. Through theoretical analysis and experimental evaluation, we show that the masked autoencoder is effective in alleviating the over-smoothing issue in deep transformer training. Based on this finding, we propose Bamboo, an idea of using deeper and narrower transformer configurations, for masked autoencoder training. On ImageNet, with such a simple change in configuration, re-designed model achieves 87.1% top-1 accuracy and outperforms SoTA models like MAE and BEiT. On language tasks, re-designed model outperforms BERT with default setting by 1.1 points on average, on GLUE datasets.