Task Matrices: Linear Maps for Cross-Model Finetuning Transfer
This work addresses the problem of efficient cross-model finetuning transfer for researchers and practitioners in machine learning, though it appears incremental as it builds on prior interpretability findings.
The paper tackled the problem of demonstrating linear representations in general adaptation regimes by developing task matrices—linear transformations from base to finetuned embedding states. The result showed that base models augmented with task matrices surpassed linear probes and sometimes approached finetuned levels across vision and text models on ten datasets.
Results in interpretability suggest that large vision and language models learn implicit linear encodings when models are biased by in-context prompting. However, the existence of similar linear representations in more general adaptation regimes has not yet been demonstrated. In this work, we develop the concept of a task matrix, a linear transformation from a base to finetuned embedding state. We demonstrate that for vision and text models and ten different datasets, a base model augmented with a task matrix achieves results surpassing linear probes, sometimes approaching finetuned levels. Our results validate the existence of cross-layer linear encodings between pretrained and finetuned architectures. Moreover, we show that a data-based approximation for such encodings is both efficient and generalizable to multiple domains. We make our implementation publicly available.