Referring to Screen Texts with Voice Assistants
This work addresses a specific usability issue for voice assistant users, offering an incremental improvement in reference understanding.
The paper tackles the problem of voice assistants' limited context understanding by enabling users to refer to texts like phone numbers or addresses on their phone screens, proposing a lightweight model that relies on extracted UI text rather than pixels.
Voice assistants help users make phone calls, send messages, create events, navigate, and do a lot more. However, assistants have limited capacity to understand their users' context. In this work, we aim to take a step in this direction. Our work dives into a new experience for users to refer to phone numbers, addresses, email addresses, URLs, and dates on their phone screens. Our focus lies in reference understanding, which becomes particularly interesting when multiple similar texts are present on screen, similar to visual grounding. We collect a dataset and propose a lightweight general-purpose model for this novel experience. Due to the high cost of consuming pixels directly, our system is designed to rely on the extracted text from the UI. Our model is modular, thus offering flexibility, improved interpretability, and efficient runtime memory utilization.