Enhancing Cross-task Transfer of Large Language Models via Activation Steering
This addresses the challenge of robust and efficient knowledge transfer for LLMs in low-resource applications, representing an incremental improvement over existing methods.
The paper tackles the problem of large language models struggling with unseen tasks in data-scarce scenarios by proposing CAST, a framework for cross-task transfer via activation steering without parameter updates, which outperforms baselines in experiments across cross-domain and cross-lingual settings.
Large language models (LLMs) have shown impressive abilities in leveraging pretrained knowledge through prompting, but they often struggle with unseen tasks, particularly in data-scarce scenarios. While cross-task in-context learning offers a direct solution for transferring knowledge across tasks, it still faces critical challenges in terms of robustness, scalability, and efficiency. In this paper, we investigate whether cross-task transfer can be achieved via latent space steering without parameter updates or input expansion. Through an analysis of activation patterns in the latent space of LLMs, we observe that the enhanced activations induced by in-context examples have consistent patterns across different tasks. Inspired by these findings, we propose CAST, a novel Cross-task Activation Steering Transfer framework that enables effective transfer by manipulating the model's internal activation states. Our approach first selects influential and diverse samples from high-resource tasks, then utilizes their contrastive representation-enhanced activations to adapt LLMs to low-resource tasks. Extensive experiments across both cross-domain and cross-lingual transfer settings show that our method outperforms competitive baselines and demonstrates superior scalability and lower computational costs.