MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop
For researchers and developers of computer use agents, this benchmark provides a more realistic and granular evaluation for macOS automation, revealing that skill libraries, not frameworks, drive performance.
MacAgentBench introduces a benchmark of 676 tasks across 25 macOS applications, with nearly 60% requiring both GUI and CLI interaction, and uses fine-grained multi-checkpoint scoring. The best configuration (Claude Opus 4.6 on OpenClaw) achieves 73.7% Pass@1, with the advantage primarily driven by the skill library rather than framework design.
Computer use agents (CUAs) have advanced rapidly in desktop automation, and a growing number of users deploy CUAs such as OpenClaw on Mac Mini for always-on automation. However, existing benchmarks, including those for macOS, evaluate agents without framework augmentation and rely on binary evaluation. As a result, they fail to capture both the framework capabilities leveraged by modern CUAs and the partial progress on long-horizon, multi-application tasks. We present MacAgentBench, a comprehensive macOS agent benchmark comprising 676 tasks across 25 applications, with nearly 60% involving both GUI and CLI interaction. The benchmark adopts deterministic rule-based evaluation and introduces fine-grained multi-checkpoint scoring with capability annotations for multi-application tasks. Experiments across three frameworks and 16 models show that the best configuration, Claude Opus 4.6 on OpenClaw, attains 73.7% Pass@1, while this advantage is primarily driven by the skill library rather than by framework design. Fine-grained metrics further reveal that models with similar Pass@1 can differ substantially in sub-goal completion. Our code and data are publicly available at https://github.com/JetAstra/MacAgentBench.