Trojan Detection Through Pattern Recognition for Large Language Models
This addresses a security threat for users of large language models, but it is incremental as it builds on existing trigger identification methods.
The paper tackles the problem of detecting Trojan backdoors in large language models, which are injected during stages like pretraining and fine-tuning, by proposing a multistage framework involving token filtration, trigger identification, and verification, and shows promising results on datasets such as TrojAI and RLHF poisoned models.
Trojan backdoors can be injected into large language models at various stages, including pretraining, fine-tuning, and in-context learning, posing a significant threat to the model's alignment. Due to the nature of causal language modeling, detecting these triggers is challenging given the vast search space. In this study, we propose a multistage framework for detecting Trojan triggers in large language models consisting of token filtration, trigger identification, and trigger verification. We discuss existing trigger identification methods and propose two variants of a black-box trigger inversion method that rely on output logits, utilizing beam search and greedy decoding respectively. We show that the verification stage is critical in the process and propose semantic-preserving prompts and special perturbations to differentiate between actual Trojan triggers and other adversarial strings that display similar characteristics. The evaluation of our approach on the TrojAI and RLHF poisoned model datasets demonstrates promising results.