Cross-Model Transfer of Task Vectors via Few-Shot Orthogonal Alignment
This work addresses a bottleneck in cross-model transfer for efficient model editing, though it is incremental as it builds on existing task arithmetic frameworks.
The paper tackles the problem of transferring task vectors between independently pre-trained models, which is limited by existing methods that assume shared initialization. It proposes a few-shot orthogonal alignment method that improves transfer accuracy over direct application and achieves performance comparable to few-shot fine-tuning on eight classification datasets.
Task arithmetic enables efficient model editing by representing task-specific changes as vectors in parameter space. Task arithmetic typically assumes that the source and target models are initialized from the same pre-trained parameters. This assumption limits its applicability in cross-model transfer settings, where models are independently pre-trained on different datasets. To address this challenge, we propose a method based on few-shot orthogonal alignment, which aligns task vectors to the parameter space of a differently pre-trained target model. These transformations preserve key properties of task vectors, such as norm and rank, and are learned using only a small number of labeled examples. We evaluate the method using two Vision Transformers pre-trained on YFCC100M and LAION400M, and test on eight classification datasets. Experimental results show that our method improves transfer accuracy over direct task vector application and achieves performance comparable to few-shot fine-tuning, while maintaining the modularity and reusability of task vectors. Our code is available at https://github.com/kawakera-lab/CrossModelTransfer.