Dataless Weight Disentanglement in Task Arithmetic via Kronecker-Factored Approximate Curvature
This addresses the issue of modular and scalable adaptation of foundation models without requiring external task data, which is crucial for privacy and data availability constraints, though it is incremental as it builds on existing regularization techniques.
The paper tackled the problem of cross-task interference in task arithmetic for adapting foundation models, which causes representation drift and degraded performance, by proposing a dataless regularization method using Kronecker-Factored Approximate Curvature, achieving state-of-the-art results in task addition and negation with constant complexity and no need for held-out tuning.
Task Arithmetic yields a modular, scalable way to adapt foundation models. Combining multiple task vectors, however, can lead to cross-task interference, causing representation drift and degraded performance. Representation drift regularization provides a natural remedy to disentangle task vectors; however, existing approaches typically require external task data, conflicting with modularity and data availability constraints (e.g., privacy requirements). We propose a dataless approach by framing regularization against representation drift as a curvature matrix approximation problem. This allows us to leverage well-established techniques; in particular, we adopt Kronecker-Factored Approximate Curvature and obtain a practical regularizer that achieves state-of-the-art results in task addition and negation. Our method has constant complexity in the number of tasks and promotes robustness to task vector rescaling, eliminating the need for held-out tuning.