A Manifold Representation of the Key in Vision Transformers
This is an incremental improvement for vision tasks like image classification, object detection, and instance segmentation.
The paper tackles the problem of intertwined query, key, and value representations in Vision Transformers by disentangling the key and adopting a manifold representation for it, resulting in performance improvements such as a 0.87% increase in top-1 accuracy for ViT-B and 0.52% for Swin-T on ImageNet-1K.
Vision Transformers implement multi-head self-attention via stacking multiple attention blocks. The query, key, and value are often intertwined and generated within those blocks via a single, shared linear transformation. This paper explores the concept of disentangling the key from the query and value, and adopting a manifold representation for the key. Our experiments reveal that decoupling and endowing the key with a manifold structure can enhance the model's performance. Specifically, ViT-B exhibits a 0.87% increase in top-1 accuracy, while Swin-T sees a boost of 0.52% in top-1 accuracy on the ImageNet-1K dataset, with eight charts in the manifold key. Our approach also yields positive results in object detection and instance segmentation tasks on the COCO dataset. We establish that these performance gains are not merely due to the simplicity of adding more parameters and computations. Future research may investigate strategies for cutting the budget of such representations and aim for further performance improvements based on our findings.