GUI vs. CLI: Execution Bottlenecks in Screen-Only and Skill-Mediated Computer-Use Agents
For researchers building computer-use agents, this work provides a controlled comparison revealing that GUI and CLI agents face distinct execution bottlenecks, with CLI deficits largely due to incomplete skill coverage.
The paper introduces a matched benchmark of 440 desktop tasks to compare GUI and CLI computer-use agents. The strongest GUI agent achieves 59.1% pass rate vs. 48.2% for CLI, but verifier-guided skill augmentation raises CLI to 69.3%, showing bottlenecks differ: GUI struggles with reliable interaction, CLI with skill coverage.
Computer-use agents can execute software tasks through either graphical interfaces or programmatic command interfaces, but existing evaluations confound interaction modality with differences in tasks, initial states, verifiers, and permitted actions. We introduce a matched execution-layer benchmark of 440 desktop tasks across 18 applications and 12 workflow categories, where screen-only GUI agents and skill-mediated CLI agents receive identical goals, states, and final-state verifiers while being restricted to modality-native actions. In this controlled setting, the strongest GUI agent reaches a 59.1% full pass rate, outperforming the strongest original-skill CLI agent at 48.2%; however, verifier-guided skill augmentation raises CLI success to 69.3%, showing that much of the CLI deficit comes from incomplete skill coverage rather than model capability alone. These results suggest that GUI and CLI expose different execution bottlenecks: GUI agents are limited by reliable grounded interaction over long-horizon workflows, whereas CLI agents are limited by the coverage and scalability of their skill interfaces.