SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment
For developers and users of LLM agent skill marketplaces, SkillAudit provides a systematic method to assess skill quality beyond fixed benchmarks, addressing the need for reliable deployment decisions.
SkillAudit introduces a framework for evaluating arbitrary LLM agent skills by automatically generating multi-dimensional reports on utility, efficiency/cost, and safety, using capability-aligned tasks and sandboxed execution. In a scan of top-ranked skills across 23 categories, over 7% were found to be risky.
Agent skills have become a practical way to extend large language model agents, but the growing skill ecosystem still lacks a reliable way to judge whether a skill is worth deploying. Existing evaluation methods remain largely anchored to fixed task suites, assessing skills through performance on predefined tasks and environments. As skill marketplaces expand, this paradigm becomes inadequate: fixed suites can conflate a skill's marginal contribution with backbone strength and miss its value when tasks fall outside the skill's intended scope. We introduce SkillAudit, an end-to-end framework for skill-centered assessment that takes an arbitrary agent skill as input and automatically generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety. SkillAudit focuses on the skill artifact itself and constructs capability-aligned evaluation tasks directly from the skill package. The generated tasks are conducted in isolated sandbox environments to collect execution evidence, followed by automated checks with LLM-based judging to produce auditable results. To dissect the agent skills, we propose the baseline comparison principle to measure utility and efficiency/cost, and introduce a two-stage detection paradigm combining static semantic analysis with dynamic runtime verification to assess safety risks. After scanning top-ranked real-world skill packages spanning 23 occupational categories, we found that over 7% of skills are at risky status.