Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
This work addresses the need for succinct UI descriptions to aid language-based applications, though it is incremental as it builds on existing multimodal techniques.
The authors tackled the problem of automatically generating language descriptions for mobile user interfaces by introducing Screen2Words, a multimodal learning approach that achieved high-quality summaries as evaluated by both automatic metrics and human ratings on a dataset of over 112k annotations across ~22k unique screens.
Mobile User Interface Summarization generates succinct language descriptions of mobile screens for conveying important contents and functionalities of the screen, which can be useful for many language-based application scenarios. We present Screen2Words, a novel screen summarization approach that automatically encapsulates essential information of a UI screen into a coherent language phrase. Summarizing mobile screens requires a holistic understanding of the multi-modal data of mobile UIs, including text, image, structures as well as UI semantics, motivating our multi-modal learning approach. We collected and analyzed a large-scale screen summarization dataset annotated by human workers. Our dataset contains more than 112k language summarization across $\sim$22k unique UI screens. We then experimented with a set of deep models with different configurations. Our evaluation of these models with both automatic accuracy metrics and human rating shows that our approach can generate high-quality summaries for mobile screens. We demonstrate potential use cases of Screen2Words and open-source our dataset and model to lay the foundations for further bridging language and user interfaces.