Beyond the final layer: Attentive multilayer fusion for vision transformers
This work addresses the challenge of adapting foundation models for downstream tasks, offering a task-aware approach that improves performance, though it is incremental as it builds on existing probing methods.
The paper tackled the problem of efficiently adapting large-scale foundation models to downstream tasks by showing that task-relevant information is distributed across network layers, not just the last layer, and introduced an attentive probing mechanism that fuses representations from all layers of a Vision Transformer. This method achieved consistent, substantial gains over standard linear probes across 20 diverse datasets and multiple pretrained models.
With the rise of large-scale foundation models, efficiently adapting them to downstream tasks remains a central challenge. Linear probing, which freezes the backbone and trains a lightweight head, is computationally efficient but often restricted to last-layer representations. We show that task-relevant information is distributed across the network hierarchy rather than solely encoded in any of the last layers. To leverage this distribution of information, we apply an attentive probing mechanism that dynamically fuses representations from all layers of a Vision Transformer. This mechanism learns to identify the most relevant layers for a target task and combines low-level structural cues with high-level semantic abstractions. Across 20 diverse datasets and multiple pretrained foundation models, our method achieves consistent, substantial gains over standard linear probes. Attention heatmaps further reveal that tasks different from the pre-training domain benefit most from intermediate representations. Overall, our findings underscore the value of intermediate layer information and demonstrate a principled, task aware approach for unlocking their potential in probing-based adaptation.