Shenghan Zheng

Xiangyi Li, Wenbo Chen, Yimin Liu et al. · berkeley

Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills. We test 7 agent-model configurations over 7,308 trajectories. Curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain (+4.5pp for Software Engineering to +51.9pp for Healthcare) and 16 of 84 tasks show negative deltas. Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming. Focused Skills with 2--3 modules outperform comprehensive documentation, and smaller models with Skills can match larger models without them.

Xiangyi Li, Kyoung Whan Choe, Yimin Liu et al. · apple-ml

Large language model (LLM) agents are increasingly deployed to automate productivity tasks (e.g., email, scheduling, document management), but evaluating them on live services is risky due to potentially irreversible changes. Existing benchmarks rely on simplified environments and fail to capture realistic, stateful, multi-service workflows. We introduce ClawsBench, a benchmark for evaluating and improving LLM agents in realistic productivity settings. It includes five high-fidelity mock services (Gmail, Slack, Google Calendar, Google Docs, Google Drive) with full state management and deterministic snapshot/restore, along with 44 structured tasks covering single-service, cross-service, and safety-critical scenarios. We decompose agent scaffolding into two independent levers (domain skills that inject API knowledge via progressive disclosure, and a meta prompt that coordinates behavior across services) and vary both to measure their separate and combined effects. Experiments across 6 models, 4 agent harnesses, and 33 conditions show that with full scaffolding, agents achieve task success rates of 39-64% but exhibit unsafe action rates of 7-33%. On OpenClaw, the top five models fall within a 10 percentage-point band on task success (53-63%), with unsafe action rates from 7% to 23% and no consistent ordering between the two metrics. We identify eight recurring patterns of unsafe behavior, including multi-step sandbox escalation and silent contract modification.

Shenghan Zheng, Qifan Zhang

AI agent protocols -- including MCP, A2A, ANP, and ACP -- enable autonomous agents to discover capabilities, delegate tasks, and compose services across trust boundaries. Despite massive deployment (MCP alone has 97M+ monthly SDK downloads), no systematic security framework for these protocols exists. We present three contributions. First, the Agent Protocol Stack, a 6-layer architectural model that defines what a complete agent protocol must specify at each layer -- analogous to ITU-T X.800 for the OSI stack. Second, the Agent-Agnostic Security Model, 11 security principles formalized as TLA+ invariants, each tagged with a property taxonomy (spec-mandated, spec-recommended, aasm-hardening, aps-completeness) that distinguishes protocol non-conformance from framework-imposed security requirements. Third, AgentConform, a two-phase conformance checker that (i)extracts normative clauses from protocol specifications into a typed Protocol~IR with explicit Protocol/Environment/Adversary action separation, (ii)compiles the IR into TLA+ models and model-checks them against AASM invariants, then (iii)replays counterexample traces against live SDK implementations to confirm findings. We introduce the Composition Safety (CS) principle: security properties that hold for individual protocols can break when protocols are composed through shared infrastructure. We demonstrate this with formal models of five protocol composition patterns, revealing cross-protocol design gaps that individual protocol analysis cannot detect. Preliminary application to representative agent protocols reveals recurrent gaps in credential lifecycle, consent enforcement, audit completeness, and composition safety. Some findings are under coordinated disclosure; full evaluation details will be released in the complete version.