OS-Marathon: Benchmarking Computer-Use Agents on Long-Horizon Repetitive Tasks
This addresses the bottleneck in evaluating agents for tedious, repetitive professional workflows, though it is incremental as it builds on existing agent methods.
The authors tackled the lack of evaluation benchmarks for computer-use agents on long-horizon repetitive tasks by introducing OS-Marathon, a benchmark with 242 tasks across 2 domains, and proposed a cost-effective few-shot demonstration method that effectively enables agents to execute workflows on larger, unseen data.
Long-horizon, repetitive workflows are common in professional settings, such as processing expense reports from receipts and entering student grades from exam papers. These tasks are often tedious for humans since they can extend to extreme lengths proportional to the size of the data to process. However, they are ideal for Computer-Use Agents (CUAs) due to their structured, recurring sub-workflows with logic that can be systematically learned. Identifying the absence of an evaluation benchmark as a primary bottleneck, we establish OS-Marathon, comprising 242 long-horizon, repetitive tasks across 2 domains to evaluate state-of-the-art (SOTA) agents. We then introduce a cost-effective method to construct a condensed demonstration using only few-shot examples to teach agents the underlying workflow logic, enabling them to execute similar workflows effectively on larger, unseen data collections. Extensive experiments demonstrate both the inherent challenges of these tasks and the effectiveness of our proposed method. Project website: https://os-marathon.github.io/.