AppAgent: Multimodal Agents as Smartphone Users
This addresses the problem of automating smartphone app usage for users or developers, though it appears incremental as it builds on existing LLM-based agent capabilities.
The paper tackles the problem of enabling LLM-based agents to operate smartphone applications by introducing a multimodal agent framework that mimics human interactions like tapping and swiping, bypassing the need for system back-end access. The agent learns through autonomous exploration or human demonstrations, and testing on 50 tasks across 10 apps showed proficiency in handling diverse high-level tasks.
Recent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework enables the agent to operate smartphone applications through a simplified action space, mimicking human-like interactions such as tapping and swiping. This novel approach bypasses the need for system back-end access, thereby broadening its applicability across diverse apps. Central to our agent's functionality is its innovative learning method. The agent learns to navigate and use new apps either through autonomous exploration or by observing human demonstrations. This process generates a knowledge base that the agent refers to for executing complex tasks across different applications. To demonstrate the practicality of our agent, we conducted extensive testing over 50 tasks in 10 different applications, including social media, email, maps, shopping, and sophisticated image editing tools. The results affirm our agent's proficiency in handling a diverse array of high-level tasks.