Foveation-Guided Dynamic Token Selection for Robust and Efficient Vision Transformers
For practitioners deploying vision transformers in noisy or adversarial environments, FDT offers a more robust and efficient alternative without explicit adversarial training.
The paper introduces the Foveated Dynamic Transformer (FDT), a vision transformer that uses foveation-guided dynamic token selection to achieve robustness to noise and adversarial attacks while improving efficiency. At 50% fixation budget, FDT achieves 81.9% accuracy (vs. 80.9% for DeiT-S) with 34.57% fewer multiply-accumulate operations.
The human visual system (HVS) employs foveated sampling and eye movements to achieve efficient perception, conserving both metabolic energy and computational resources. Drawing inspiration from this robustness and adaptability, we introduce the Foveated Dynamic Transformer (FDT), a foveation-guided dynamic token-selection architecture that integrates these mechanisms into a vision transformer framework. The FDT exhibits strong resilience to various types of noise and adversarial attacks, despite not being explicitly trained for such challenges. This inherent robustness is achieved through the use of fixation and foveation modules: the fixation module identifies fixation points to filter out irrelevant information, while the foveation module generates foveated embeddings with multi-scale information. At the 50% fixation-budget setting, FDT achieves higher accuracy than DeiT-S (81.9% vs. 80.9%) while reducing multiply-accumulate operations by 34.57%, highlighting one operating point on its accuracy-efficiency trade-off. These attributes position FDT as an HVS-inspired step toward artificial neural networks that combine adaptive computation with improved resilience.