Vision-TTT: Efficient and Expressive Visual Representation Learning with Test-Time Training
This addresses efficiency and scalability issues for computer vision applications, offering a potential next-generation visual backbone, though it appears incremental as an extension of existing test-time training methods.
The paper tackles the quadratic complexity of Vision Transformers by introducing Vision-TTT, a linear-time method that compresses visual token sequences using self-supervised learning, achieving up to 82.5% Top-1 accuracy on ImageNet and reducing FLOPs by 79.4% with 4.38x faster runtime compared to DeiT-T.
Learning efficient and expressive visual representation has long been the pursuit of computer vision research. While Vision Transformers (ViTs) gradually replace traditional Convolutional Neural Networks (CNNs) as more scalable vision learners, their applications are plagued by the quadratic complexity of the self-attention mechanism. To address the challenge, we introduce a new linear-time sequence modeling method Test-Time Training (TTT) into vision and propose Vision-TTT, which compresses the visual token sequence in a novel self-supervised learning manner. By incorporating bidirectional scan strategy and the Conv2d module, Vision-TTT effectively extends vanilla TTT to model 2D visual correlations with global receptive fields. Extensive experiments show that \texttt{Vittt-T/S/B} achieve 77.3%,81.2%,82.5% Top-1 accuracy on ImageNet classification and also greatly outperform their counterparts on downstream tasks. At 1280x1280 resolution, \texttt{Vittt-T} reduces FLOPs by 79.4% and runs 4.38x faster with 88.9% less memory than DeiT-T. These results demonstrate the expressiveness and efficiency of Vision-TTT as a strong candidate for the next-generation generic visual backbone.