SkillSentry: Adaptive Honey Worlds for Dynamic Safety Testing of Agent Skills
For developers and security researchers, this provides a more effective method to detect hidden unsafe behaviors in LLM agent skills, addressing a critical security gap.
SkillSentry is a dynamic safety-testing framework for LLM agent skills that uses adaptive honey worlds to elicit and attribute conditional harmful behaviors. It achieves 99.50% recall and 96.26% average F1 on standard benchmarks, and 92.95% average F1 under evasion, outperforming baselines.
External skills extend the capabilities of large language model agents, but also introduce an execution-time attack surface: a skill that appears benign under inspection may reveal harmful behavior only after particular environmental states, resources, or interaction histories are encountered. Existing scanners primarily rely on static analysis, predefined rules, or one-shot semantic judgments, making such conditional behavior difficult to elicit and attribute. We present SkillSentry, a dynamic safety-testing framework based on adaptive honey worlds. SkillSentry infers the intended capability boundary of a skill, constructs an LLM-simulated environment with controlled decoy resources, and adaptively generates tasks to explore its behavioral states. It then compares skill-enabled trajectories with matched no-skill executions, grounding suspicious behaviors in source code and verified execution traces before making a final decision. We evaluate SkillSentry against seven scanner configurations. SkillSentry achieves 99.50% Recall and 96.26% average F1 on standard benchmarks. Under semantics-preserving evasion, it reaches 92.95% average F1, compared with 80.07% for the strongest baselines. Our code is available at https://github.com/nizhangli062-jpg/SkillSentry-Adaptive-Honey-Worlds-for-Dynamic-Safety-Testing-of-Agent-Skills.