Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers
This addresses the problem of applying vision transformers to real-world scenarios like active visual exploration where input patches vary in scale and position.
The paper tackles the limitation of rigid input sampling in vision transformers by formalizing input elasticity and proposing architectural modifications to increase it, achieving improved performance on elastic sampling benchmarks.
Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems, such as active visual exploration, where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover, we propose modifications to the transformer architecture and training regime, which increase its elasticity. Through extensive experimentation, we spotlight opportunities and challenges associated with such architecture.