Autonomous Continual Learning of Computer-Use Agents for Environment Adaptation
This work addresses the challenge of continual learning for computer-use agents in specific environments, offering an incremental improvement by automating task generation and evaluation to reduce reliance on costly human data.
The paper tackles the problem of enabling computer-use agents to adapt to diverse and dynamic digital environments without human annotation, achieving 4-22% performance gains without catastrophic forgetting. It introduces ACuRL, an autonomous curriculum reinforcement learning framework with a robust evaluator that reaches 93% agreement with human judgments.
Real-world digital environments are highly diverse and dynamic. These characteristics cause agents to frequently encounter unseen scenarios and distribution shifts, making continual learning in specific environments essential for computer-use agents (CUAs). However, a key challenge lies in obtaining high-quality and environment-grounded agent data without relying on costly human annotation. In this work, we introduce ACuRL, an Autonomous Curriculum Reinforcement Learning framework that continually adapts agents to specific environments with zero human data. The agent first explores target environments to acquire initial experiences. During subsequent iterative training, a curriculum task generator leverages these experiences together with feedback from the previous iteration to synthesize new tasks tailored for the agent's current capabilities. To provide reliable reward signals, we introduce CUAJudge, a robust automatic evaluator for CUAs that achieves 93% agreement with human judgments. Empirically, our method effectively enables both intra-environment and cross-environment continual learning, yielding 4-22% performance gains without catastrophic forgetting on existing environments. Further analyses show highly sparse updates (e.g., 20% parameters), which helps explain the effective and robust adaptation. Our data and code are available at https://github.com/OSU-NLP-Group/ACuRL.