Enabling Conversational Interaction with Mobile UI using Large Language Models
This addresses the challenge of expensive and effort-consuming development for language-based mobile interaction, offering a lightweight and generalizable approach.
The paper tackled the problem of enabling versatile conversational interactions with mobile UIs by using a single large language model with prompting techniques, achieving competitive performance on four modeling tasks without dedicated datasets or training.
Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task, which is expensive and effort-consuming. Recently, pre-trained large language models (LLMs) have been shown capable of generalizing to various downstream tasks when prompted with a handful of examples from the target task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single LLM. We designed prompting techniques to adapt an LLM to mobile UIs. We experimented with four important modeling tasks that address various scenarios in conversational interaction. Our method achieved competitive performance on these challenging tasks without requiring dedicated datasets and training, offering a lightweight and generalizable approach to enable language-based mobile interaction.