Dual PatchNorm
This is an incremental improvement for vision transformer researchers, addressing a specific architectural bottleneck.
The paper tackles the problem of optimizing LayerNorm placement in Vision Transformers by proposing Dual PatchNorm, which adds two LayerNorm layers before and after the patch embedding layer. This simple modification consistently improves accuracy over well-tuned models without negative effects.
We propose Dual PatchNorm: two Layer Normalization layers (LayerNorms), before and after the patch embedding layer in Vision Transformers. We demonstrate that Dual PatchNorm outperforms the result of exhaustive search for alternative LayerNorm placement strategies in the Transformer block itself. In our experiments, incorporating this trivial modification, often leads to improved accuracy over well-tuned Vision Transformers and never hurts.