CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planning
This work addresses inefficiencies in mobile assistant task decomposition for users, but it is incremental as it builds on existing VLM-based architectures.
The paper tackles the problem of ineffective and inefficient subtask planning in visual language model-based mobile operating assistants by proposing a new architecture called CHOP, which uses human-planned subtasks as a basis to improve planning in GUI scenarios, resulting in significant improvements in effectiveness and efficiency across 20 apps in English and Chinese contexts.
The advancement of visual language models (VLMs) has enhanced mobile device operations, allowing simulated human-like actions to address user requirements. Current VLM-based mobile operating assistants can be structured into three levels: task, subtask, and action. The subtask level, linking high-level goals with low-level executable actions, is crucial for task completion but faces two challenges: ineffective subtasks that lower-level agent cannot execute and inefficient subtasks that fail to contribute to the completion of the higher-level task. These challenges stem from VLM's lack of experience in decomposing subtasks within GUI scenarios in multi-agent architecture. To address these, we propose a new mobile assistant architecture with constrained high-frequency o}ptimized planning (CHOP). Our approach overcomes the VLM's deficiency in GUI scenarios planning by using human-planned subtasks as the basis vector. We evaluate our architecture in both English and Chinese contexts across 20 Apps, demonstrating significant improvements in both effectiveness and efficiency. Our dataset and code is available at https://github.com/Yuqi-Zhou/CHOP