Comparing Human Oversight Strategies for Computer-Use Agents
This work addresses the challenge of designing effective oversight for computer-use agents, which is important for users and developers, but it is incremental as it builds on existing oversight mechanisms by providing a comparative framework.
The study tackled the problem of comparing human oversight strategies for LLM-powered computer-use agents by conceptualizing oversight as a structural coordination problem and testing four strategies with 48 participants in a live web environment. Results showed that oversight strategy more reliably shaped users' exposure to problematic actions than their ability to correct them, with plan-based strategies associated with lower rates of problematic-action occurrence but not equally strong gains in runtime intervention success.
LLM-powered computer-use agents (CUAs) are shifting users from direct manipulation to supervisory coordination. Existing oversight mechanisms, however, have largely been studied as isolated interface features, making broader oversight strategies difficult to compare. We conceptualize CUA oversight as a structural coordination problem defined by delegation structure and engagement level, and use this lens to compare four oversight strategies in a mixed-methods study with 48 participants in a live web environment. Our results show that oversight strategy more reliably shaped users' exposure to problematic actions than their ability to correct them once visible. Plan-based strategies were associated with lower rates of agent problematic-action occurrence, but not equally strong gains in runtime intervention success once such actions became visible. On subjective measures, no single strategy was uniformly best, and the clearest context-sensitive differences appeared in trust. Qualitative findings further suggest that intervention depended not only on what controls users retained, but on whether risky moments became legible as requiring judgment during execution. These findings suggest that effective CUA oversight is not achieved by maximizing human involvement alone. Instead, it depends on how supervision is structured to surface decision-critical moments and support their recognition in time for meaningful intervention.