SCUBA: Salesforce Computer Use Benchmark
This addresses the need for realistic evaluation of computer-use agents in enterprise software ecosystems like Salesforce, though it appears incremental as it builds on existing benchmarking approaches.
The researchers tackled the problem of evaluating computer-use agents on customer relationship management workflows by introducing SCUBA, a benchmark with 300 task instances derived from real user interviews on the Salesforce platform. They found that in zero-shot settings, open-source model agents achieved less than 5% success rate while closed-source models reached up to 39%, and demonstration-augmented settings improved success rates to 50% while reducing time and costs by 13% and 16% respectively.
We introduce SCUBA, a benchmark designed to evaluate computer-use agents on customer relationship management (CRM) workflows within the Salesforce platform. SCUBA contains 300 task instances derived from real user interviews, spanning three primary personas, platform administrators, sales representatives, and service agents. The tasks test a range of enterprise-critical abilities, including Enterprise Software UI navigation, data manipulation, workflow automation, information retrieval, and troubleshooting. To ensure realism, SCUBA operates in Salesforce sandbox environments with support for parallel execution and fine-grained evaluation metrics to capture milestone progress. We benchmark a diverse set of agents under both zero-shot and demonstration-augmented settings. We observed huge performance gaps in different agent design paradigms and gaps between the open-source model and the closed-source model. In the zero-shot setting, open-source model powered computer-use agents that have strong performance on related benchmarks like OSWorld only have less than 5\% success rate on SCUBA, while methods built on closed-source models can still have up to 39% task success rate. In the demonstration-augmented settings, task success rates can be improved to 50\% while simultaneously reducing time and costs by 13% and 16%, respectively. These findings highlight both the challenges of enterprise tasks automation and the promise of agentic solutions. By offering a realistic benchmark with interpretable evaluation, SCUBA aims to accelerate progress in building reliable computer-use agents for complex business software ecosystems.