Gradient-Sign Masking for Task Vector Transport Across Pre-Trained Models
This addresses the need for efficient adaptation of foundation models to specific tasks without full fine-tuning, though it is incremental as it builds on existing task vector concepts.
The paper tackles the problem of transferring task vectors across different pre-trained models, which often fails due to misaligned parameter spaces, and proposes GradFix, a method that uses gradient sign masking to achieve this transfer with only a few labeled samples, resulting in significant performance gains on vision and language benchmarks.
When a new release of a foundation model is published, practitioners typically need to repeat full fine-tuning, even if the same task has already been solved in the previous version. A promising alternative is to reuse the parameter changes (i.e., task vectors) that capture how a model adapts to a specific task. However, they often fail to transfer across different pre-trained models due to their misaligned parameter space. In this work, we show that the key to successful transfer lies in the sign structure of the gradients of the new model. Based on this insight, we propose GradFix, a novel method that approximates the ideal gradient sign structure and leverages it to transfer knowledge using only a handful of labeled samples. Notably, this requires no additional fine-tuning: the adaptation is achieved by computing a few gradients at the target model and masking the source task vector accordingly. This yields an update that is locally aligned with the target loss landscape, effectively rebasing the task vector onto the new pre-training. We provide a theoretical guarantee that our method ensures first-order descent. Empirically, we demonstrate significant performance gains on vision and language benchmarks, consistently outperforming naive task vector addition and few-shot fine-tuning.