Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers
For practitioners deploying ViTs, Fusion provides a unified adaptive inference framework that outperforms prior methods in accuracy-latency trade-offs and calibration, with transferable performance across datasets.
Fusion coordinates token merging, early exiting, and token pruning in a staged design to reduce computation in Vision Transformers without accuracy loss, achieving up to 4× lower calibration error and 48% inference energy reduction on ImageNet-1k with DeiT-S.
Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative. Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early, but combining these mechanisms often causes unstable intermediate representations and accuracy degradation. We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning through a simple staged design: tokens are merged first, confidence is evaluated next, and pruning is applied only to samples that continue inference. This ordering allows the three mechanisms to operate cooperatively rather than competitively. Fusion further includes lightweight routing modules that adapt compression strength to each input and support inference-time adjustment of the accuracy--latency trade-off without retraining. On ImageNet-1k with DeiT-S, Fusion matches or surpasses state-of-the-art adaptive ViT methods at comparable compute budgets while reducing calibration error by up to $4\times$ and inference energy by $48\%$. Experiments across ImageNet-100, CIFAR-100, and ImageNette with multiple ViT backbones demonstrate consistent transferability without dataset-specific tuning.