Res-Attn : An Enhanced Res-Tuning Approach with Lightweight Attention Mechanism
This work addresses the need for flexible and efficient model adaptation, offering an incremental improvement over prior Res-Tuning approaches.
The paper tackles the problem of efficiently tuning pre-trained models for new scenarios by proposing Res-Attn, an enhanced tuner with a lightweight attention mechanism, achieving superior performance in discriminative and generative tasks compared to existing methods.
Res-Tuning introduces a flexible and efficient paradigm for model tuning, showing that tuners decoupled from the backbone network can achieve performance comparable to traditional methods. Existing methods commonly construct the tuner as a set of trainable low-rank decomposition matrices, positing that a low-rank subspace suffices for adapting pre-trained foundational models to new scenarios. In this work, we present an advanced, efficient tuner augmented with low-rank attention, termed Res-Attn , which also adheres to the Res-Tuning framework. Res-Attn utilizes a parallel multi-head attention module equipped with low-rank projections for query, key, and value to execute streamlined attention operations. Through training this lightweight attention module, Res-Attn facilitates adaptation to new scenarios. Our extensive experiments across a range of discriminative and generative tasks showcase the superior performance of our method when compared to existing alternatives