On Separate Normalization in Self-supervised Transformers
This work addresses a specific bottleneck in transformer-based models for researchers and practitioners, but it is incremental as it modifies an existing normalization approach.
The paper tackled the problem of using a single normalization layer for both tokens and the [CLS] symbol in self-supervised transformers, which may not optimally align with their distinct roles, and found that employing separate normalization layers resulted in an average 2.7% performance improvement across image, natural language, and graph domains.
Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both the [CLS] symbol and the tokens. We propose in this paper a simple modification that employs separate normalization layers for the tokens and the [CLS] symbol to better capture their distinct characteristics and enhance downstream task performance. Our method aims to alleviate the potential negative effects of using the same normalization statistics for both token types, which may not be optimally aligned with their individual roles. We empirically show that by utilizing a separate normalization layer, the [CLS] embeddings can better encode the global contextual information and are distributed more uniformly in its anisotropic space. When replacing the conventional normalization layer with the two separate layers, we observe an average 2.7% performance improvement over the image, natural language, and graph domains.