Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learning
This addresses efficiency bottlenecks in vision transformers for computer vision applications, though it appears incremental in its technical approach.
The paper investigates how many visual tokens are needed to preserve image semantics in vision transformers, finding that larger models require significantly fewer tokens. It proposes Orthogonal Filtering to cluster redundant tokens into compact orthogonal bases, achieving up to 40% token reduction with minimal accuracy loss.
This paper investigates the fundamental relationship between model capacity and the minimal number of visual tokens required to preserve image semantics. Inspired by the Minimum Description Length principle, we reinterpret image tokens as vectors in a visual semantic space and define the intrinsic semantic complexity of an image as the smallest set of basis vectors needed to span this space. Building on this perspective, we propose Orthogonal Filtering, a lightweight module that adaptively clusters redundant tokens into a compact set of orthogonal bases. Through extensive experiments across a range of ViT models, we reveal a consistent token, model scaling law: larger models require significantly fewer tokens to span visual semantic space. Besides, we also contribute a visual long-context dataset.