Agent-SAMA: State-Aware Mobile Assistant
This addresses the robustness problem for mobile GUI agents by providing a structured state modeling approach, though it appears incremental as it builds on existing MLLM-based agents.
The paper tackles the problem of mobile GUI agents being reactive and lacking structured understanding of app navigation flows, which limits their ability to handle execution context, detect errors, and recover. The result is Agent-SAMA, a state-aware multi-agent framework that models app execution as a Finite State Machine, achieving success rates up to 84.0% and recovery rates up to 71.9% on benchmarks, with improvements of up to 12% in task success and 13.8% in recovery success over prior methods.
Mobile Graphical User Interface (GUI) agents aim to autonomously complete tasks within or across apps based on user instructions. While recent Multimodal Large Language Models (MLLMs) enable these agents to interpret UI screens and perform actions, existing agents remain fundamentally reactive. They reason over the current UI screen but lack a structured representation of the app navigation flow, limiting GUI agents' ability to understand execution context, detect unexpected execution results, and recover from errors. We introduce Agent-SAMA, a state-aware multi-agent framework that models app execution as a Finite State Machine (FSM), treating UI screens as states and user actions as transitions. Agent-SAMA implements four specialized agents that collaboratively construct and use FSMs in real time to guide task planning, execution verification, and recovery. We evaluate Agent-SAMA on two types of benchmarks: cross-app (Mobile-Eval-E, SPA-Bench) and mostly single-app (AndroidWorld). On Mobile-Eval-E, Agent-SAMA achieves an 84.0% success rate and a 71.9% recovery rate. On SPA-Bench, it reaches an 80.0% success rate with a 66.7% recovery rate. Compared to prior methods, Agent-SAMA improves task success by up to 12% and recovery success by 13.8%. On AndroidWorld, Agent-SAMA achieves a 63.7% success rate, outperforming the baselines. Our results demonstrate that structured state modeling enhances robustness and can serve as a lightweight, model-agnostic memory layer for future GUI agents.