CVJun 30

What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States

arXiv:2606.3161222.6
Predicted impact top 4% in CV · last 90 daysOriginality Incremental advance
AI Analysis

For developers of mobile GUI agents, this work addresses the bottleneck of unreliable memory in long-horizon tasks, offering a novel method that improves execution accuracy.

Existing GUI agent memory methods treat memory as passive storage, causing unreliable task execution in long-horizon tasks. The proposed Active Task Driving Memory (ATMem) shifts to an actively maintained execution state, and STR-GRPO reinforcement learning selectively uses memory, achieving improved task completion while avoiding out-of-scope actions.

Mobile GUI agents increasingly face long-horizon tasks that require reading, updating, and reusing task-relevant data across pages and applications. Existing memory methods treat memory largely as passive storage, where past observations are accumulated and retrieved when needed. Yet retrieving a value does not reveal its current role in the workflow. The agent must still infer from accumulated records whether the value should be used now, has already been used, or must wait for a later dependency. This implicit reconstruction becomes unreliable in long trajectories with similar fields, repeated values, distractors, and outdated states, causing repeated or missed operations. We propose Active Task Driving Memory (ATMem), which shifts GUI-agent memory from passive storage to an actively maintained execution state. ATMem maintains task-relevant information as a continually updated execution state that links each value to its role and current status, enabling action selection based on the current workflow state. We therefore introduce \textbf{STR-GRPO}, an online reinforcement learning method that learns to use ATMem selectively according to its contribution to task completion. STR-GRPO contrasts memory-on and memory-off rollouts to estimate when memory use improves execution, while memory-cost-aware reward discourages costly memory usage that does not improve execution. To evaluate whether agents can complete all in-scope work while avoiding out-of-scope actions over long-horizon execution, we build a challenging mobile benchmark. From a list of near identical entries, agents must act on every entry that satisfies the instruction and reject entries that violate its constraints.

Foundations

The foundational work for this paper's niche, ranked by how specifically the neighbourhood builds on it — not by global fame.

Your Notes