Vision Pair Learning: An Efficient Training Framework for Image Classification
This work addresses the efficiency and performance gap between transformers and CNNs in vision tasks, offering a novel training framework that is incremental in combining existing architectures.
The paper tackles the problem of improving transformer performance in image classification by proposing Vision Pair Learning (VPL), a framework that pairs transformer and CNN branches with a multi-stage training strategy, resulting in top-1 accuracy of 83.47% for ViT-Base and 79.61% for ResNet-50 on ImageNet-1k.
Transformer is a potentially powerful architecture for vision tasks. Although equipped with more parameters and attention mechanism, its performance is not as dominant as CNN currently. CNN is usually computationally cheaper and still the leading competitor in various vision tasks. One research direction is to adopt the successful ideas of CNN and improve transformer, but it often relies on elaborated and heuristic network design. Observing that transformer and CNN are complementary in representation learning and convergence speed, we propose an efficient training framework called Vision Pair Learning (VPL) for image classification task. VPL builds up a network composed of a transformer branch, a CNN branch and pair learning module. With multi-stage training strategy, VPL enables the branches to learn from their partners during the appropriate stage of the training process, and makes them both achieve better performance with less time cost. Without external data, VPL promotes the top-1 accuracy of ViT-Base and ResNet-50 on the ImageNet-1k validation set to 83.47% and 79.61% respectively. Experiments on other datasets of various domains prove the efficacy of VPL and suggest that transformer performs better when paired with the differently structured CNN in VPL. we also analyze the importance of components through ablation study.