SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement
This addresses security risks in open-registry agent ecosystems for developers and users, though it is incremental as it builds on existing red-teaming and adversarial prompting methods.
The paper tackled the problem of latent vulnerabilities in non-malicious LLM-based agent skills that can be exploited through adversarial prompting without modifying the skill, and introduced SkillAttack, a red-teaming framework that achieved attack success rates of 0.73-0.93 on adversarial skills and up to 0.26 on real-world skills.
LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities, yet the openness of such ecosystems makes skills difficult to thoroughly vet. Existing attacks rely on injecting malicious instructions into skills, making them easily detectable by static auditing. However, non-malicious skills may also harbor latent vulnerabilities that an attacker can exploit solely through adversarial prompting, without modifying the skill itself. We introduce SkillAttack, a red-teaming framework that dynamically verifies skill vulnerability exploitability through adversarial prompting. SkillAttack combines vulnerability analysis, surface-parallel attack generation, and feedback-driven exploit refinement into a closed-loop search that progressively converges toward successful exploitation. Experiments across 10 LLMs on 71 adversarial and 100 real-world skills show that SkillAttack outperforms all baselines by a wide margin (ASR 0.73--0.93 on adversarial skills, up to 0.26 on real-world skills), revealing that even well-intended skills pose serious security risks under realistic agent interactions.