Lightweight Neural App Control
This addresses the challenge of real-time app control on smartphones for users, though it is incremental as it builds on existing vision-language models.
The paper tackles the problem of efficient mobile phone control by introducing LiMAC, a lightweight architecture that uses a small Action Transformer and fine-tuned vision-language model to generate precise actions from textual goals and past observations, achieving up to 19% higher action accuracy than fine-tuned VLMs and up to 42% higher than prompt-engineering baselines.
This paper introduces a novel mobile phone control architecture, Lightweight Multi-modal App Control (LiMAC), for efficient interactions and control across various Android apps. LiMAC takes as input a textual goal and a sequence of past mobile observations, such as screenshots and corresponding UI trees, to generate precise actions. To address the computational constraints inherent to smartphones, we introduce a small Action Transformer (AcT) integrated with a fine-tuned vision-language model (VLM) for real-time decision-making and task execution. We evaluate LiMAC on two open-source mobile control datasets, demonstrating the superior performance of our small-form-factor approach against fine-tuned versions of open-source VLMs, such as Florence2 and Qwen2-VL. It also significantly outperforms prompt engineering baselines utilising closed-source foundation models like GPT-4o. More specifically, LiMAC increases the overall action accuracy by up to 19% compared to fine-tuned VLMs, and up to 42% compared to prompt-engineering baselines.