ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations
This addresses the need for better UI understanding in AI systems, offering a scalable data generation method, but it is incremental as it adapts existing techniques to a new domain.
The paper tackled the problem of poor performance of multimodal vision-language models on UI tasks due to lack of training data by generating a dataset of 335K conversational examples paired with UIs without human annotations, and fine-tuned a conversational VLM that showed applicability to UI element detection, response quality, and multi-step navigation.
Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused understanding of images and language, but many perform poorly on UI tasks due to the lack of UI training data. In this paper, we adapt a recipe for generating paired text-image training data for VLMs to the UI domain by combining existing pixel-based methods with a Large Language Model (LLM). Unlike prior art, our method requires no human-provided annotations, and it can be applied to any dataset of UI screenshots. We generate a dataset of 335K conversational examples paired with UIs that cover Q&A, UI descriptions, and planning, and use it to fine-tune a conversational VLM for UI tasks. To assess the performance of our model, we benchmark it on UI element detection tasks, evaluate response quality, and showcase its applicability to multi-step UI navigation and planning.