FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents
This work addresses the problem of building more user-oriented mobile GUI agents for enhancing human-device interaction efficiency, though it is incremental as it builds on existing MLLM paradigms with new benchmarks and data.
The paper tackles the limitations of current mobile GUI agents, which lack proactive intent anticipation and personalization, by introducing the FingerTip benchmark with two new tracks for proactive task suggestions and personalized task execution, based on collected human demonstrations from real-life Android usage, and shows that fine-tuned models effectively utilize user information to achieve good results.
Mobile GUI agents are becoming critical tools for enhancing human-device interaction efficiency, with multimodal large language models (MLLMs) emerging as dominant paradigms in this domain. Current agents, however, are limited to following explicit human instructions, resulting in insufficient capability for proactive intent anticipation. Additionally, these agents fail to leverage the contextual information associated with users during task execution, thereby neglecting potentially vast differences in user preferences. To address these challenges, we introduce the FingerTip benchmark. It contains two new tracks: proactive task suggestions by analyzing environment observation and users' previous intents, and personalized task execution by catering to users' action preferences. We collected unique human demonstrations of multi-step Android device interactions across a variety of everyday apps. These demonstrations are not isolated but are continuously acquired from the users' long-term usage in their real lives, and encompass essential user-related contextual information. Our experiments reveal challenges of the tasks we propose. The model fine-tuned with the data we collected effectively utilized user information and achieved good results, highlighting the potential of our approach in building more user-oriented mobile GUI agents. Our code is open-source at https://anonymous.4open.science/r/FingerTip-57B8 for reproducibility.