DPT: Deformable Patch-based Transformer for Visual Recognition
This addresses a key bottleneck in vision transformers for computer vision tasks, offering a plug-and-play improvement for image classification and object detection.
The paper tackles the problem of fixed-size patch embeddings in vision transformers, which can destroy object semantics, by proposing a Deformable Patch module that adaptively splits images into patches with varying positions and scales in a data-driven way. Results show DPT achieves 81.9% top-1 accuracy on ImageNet classification and up to 44.3% box mAP on MSCOCO object detection.
Transformer has achieved great success in computer vision, while how to split patches in an image remains a problem. Existing methods usually use a fixed-size patch embedding which might destroy the semantics of objects. To address this problem, we propose a new Deformable Patch (DePatch) module which learns to adaptively split the images into patches with different positions and scales in a data-driven way rather than using predefined fixed patches. In this way, our method can well preserve the semantics in patches. The DePatch module can work as a plug-and-play module, which can easily be incorporated into different transformers to achieve an end-to-end training. We term this DePatch-embedded transformer as Deformable Patch-based Transformer (DPT) and conduct extensive evaluations of DPT on image classification and object detection. Results show DPT can achieve 81.9% top-1 accuracy on ImageNet classification, and 43.7% box mAP with RetinaNet, 44.3% with Mask R-CNN on MSCOCO object detection. Code has been made available at: https://github.com/CASIA-IVA-Lab/DPT .