DART: Differentiable Dynamic Adaptive Region Tokenizer for Vision Foundation Models
This addresses a fundamental efficiency problem in vision foundation models for applications like multimodal AI and robotics, representing a novel method rather than an incremental improvement.
The paper tackles the performance bottleneck of fixed-grid tokenizers in vision models by introducing DART, a differentiable dynamic adaptive region tokenizer that creates content-aware patches, resulting in a DART-equipped DeiT-Small (22M parameters) matching DeiT-Base (86M) performance with nearly double the inference speed.
The content-agnostic, fixed-grid tokenizers used by standard large-scale vision models like Vision Transformer (ViT) and Vision Mamba (Vim) represent a fundamental performance bottleneck, creating a trade-off between capturing fine-grained detail and suffering from redundant computation. To resolve this dilemma, we introduce DART, a fully differentiable Dynamic Adaptive Region Tokenizer. DART employs learnable region scores and quantile-based partitioning to create content-aware patches of varying sizes, intelligently allocating a higher token density to information-rich regions. The impact of this approach is profound: it unlocks a more intelligent scaling paradigm, where a DART-equipped DeiT-Small (22M parameters) matches the performance of a DeiT-Base (86M) with nearly double the inference speed by efficiently capturing high-resolution details in key regions. Furthermore, the principle of adaptive tokenization proves its generality with clear benefits in dense prediction and spatiotemporal video tasks. We argue that by resolving the tokenizer bottleneck at its source, adaptive tokenization is a key component for building the next generation of more efficient and capable foundation models for multimodal AI, robotics, and content generation. Code is available at https://github.com/HCPLab-SYSU/DART.