Mitigating Many-Shot Jailbreaking
This addresses a security vulnerability in LLMs for users and developers, but the approach is incremental as it builds on existing mitigation methods.
The paper tackles the problem of many-shot jailbreaking (MSJ), where attackers use long context windows to bypass LLM safety training with fake examples, and finds that combining fine-tuning and input sanitization techniques significantly reduces MSJ effectiveness while maintaining model performance.
Many-shot jailbreaking (MSJ) is an adversarial technique that exploits the long context windows of modern LLMs to circumvent model safety training by including in the prompt many examples of a "fake" assistant responding inappropriately before the final request. With enough examples, the model's in-context learning abilities override its safety training, and it responds as if it were the "fake" assistant. In this work, we probe the effectiveness of different fine-tuning and input sanitization approaches on mitigating MSJ attacks, alone and in combination. We find incremental mitigation effectiveness for each, and show that the combined techniques significantly reduce the effectiveness of MSJ attacks, while retaining model performance in benign in-context learning and conversational tasks. We suggest that our approach could meaningfully ameliorate this vulnerability if incorporated into model safety post-training.