ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks
This addresses a critical security problem for users of large language models, particularly in scenarios involving in-context learning, by providing a novel defense against backdoor attacks, though it is incremental in building upon existing security research.
The paper tackles the vulnerability of large language models to backdoor attacks in in-context learning by proposing a defense mechanism called ICLShield, which dynamically adjusts concept preferences to mitigate attacks, achieving state-of-the-art effectiveness with an average improvement of +26.02% over existing methods.
In-context learning (ICL) has demonstrated remarkable success in large language models (LLMs) due to its adaptability and parameter-free nature. However, it also introduces a critical vulnerability to backdoor attacks, where adversaries can manipulate LLM behaviors by simply poisoning a few ICL demonstrations. In this paper, we propose, for the first time, the dual-learning hypothesis, which posits that LLMs simultaneously learn both the task-relevant latent concepts and backdoor latent concepts within poisoned demonstrations, jointly influencing the probability of model outputs. Through theoretical analysis, we derive an upper bound for ICL backdoor effects, revealing that the vulnerability is dominated by the concept preference ratio between the task and the backdoor. Motivated by these findings, we propose ICLShield, a defense mechanism that dynamically adjusts the concept preference ratio. Our method encourages LLMs to select clean demonstrations during the ICL phase by leveraging confidence and similarity scores, effectively mitigating susceptibility to backdoor attacks. Extensive experiments across multiple LLMs and tasks demonstrate that our method achieves state-of-the-art defense effectiveness, significantly outperforming existing approaches (+26.02% on average). Furthermore, our method exhibits exceptional adaptability and defensive performance even for closed-source models (e.g., GPT-4).