Cross-Axis Transformer with 3D Rotary Positional Embeddings
This addresses a bottleneck in vision transformers for researchers and practitioners, offering a more efficient and effective alternative.
The paper tackles the computational inefficiency and poor spatial handling of Vision Transformers by introducing the Cross-Axis Transformer, which reduces floating point operations and achieves faster convergence and higher accuracy than existing models.
Despite lagging behind their modal cousins in many respects, Vision Transformers have provided an interesting opportunity to bridge the gap between sequence modeling and image modeling. Up until now however, vision transformers have largely been held back, due to both computational inefficiency, and lack of proper handling of spatial dimensions. In this paper, we introduce the Cross-Axis Transformer. CAT is a model inspired by both Axial Transformers, and Microsoft's recent Retentive Network, that drastically reduces the required number of floating point operations required to process an image, while simultaneously converging faster and more accurately than the Vision Transformers it replaces.