Vision Transformer Finetuning Benefits from Non-Smooth Components
This provides a novel perspective for practitioners on optimizing vision transformer finetuning, challenging the assumption that smoothness is always desirable.
The paper tackles the role of transformer smoothness in transfer learning by analyzing plasticity, showing that high plasticity (low smoothness) in attention and feedforward layers leads to better finetuning performance, with consistent gains in accuracy across datasets.
The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their plasticity. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies low smoothness. We demonstrate through theoretical analysis and comprehensive experiments that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on the functional properties of transformers. The code is available at https://github.com/ambroiseodt/vit-plasticity.