A Case for Declarative LLM-friendly Interfaces for Improved Efficiency of Computer-Use Agents
This addresses the problem of low success rates and excessive LLM calls for developers and users of computer-use agents, representing a novel method rather than an incremental improvement.
The paper tackles the inefficiency of computer-use agents on graphical user interfaces by proposing Goal-Oriented Interface (GOI), which transforms GUIs into declarative primitives, improving task success rates by 67% and reducing interaction steps by 43.5% in Microsoft Office tasks.
Computer-use agents (CUAs) powered by large language models (LLMs) have emerged as a promising approach to automating computer tasks, yet they struggle with graphical user interfaces (GUIs). GUIs, designed for humans, force LLMs to decompose high-level goals into lengthy, error-prone sequences of fine-grained actions, resulting in low success rates and an excessive number of LLM calls. We propose Goal-Oriented Interface (GOI), a novel abstraction that transforms existing GUIs into three declarative primitives: access, state, and observation, which are better suited for LLMs. Our key idea is policy-mechanism separation: LLMs focus on high-level semantic planning (policy) while GOI handles low-level navigation and interaction (mechanism). GOI does not require modifying the application source code or relying on application programming interfaces (APIs). We evaluate GOI with Microsoft Office Suite (Word, PowerPoint, Excel) on Windows. Compared to a leading GUI-based agent baseline, GOI improves task success rates by 67% and reduces interaction steps by 43.5%. Notably, GOI completes over 61% of successful tasks with a single LLM call.