A Multimodal GUI Architecture for Interfacing with LLM-Based Conversational Assistants
This work addresses the problem of making existing production applications accessible via speech for users, though it is incremental as it builds on existing protocols and patterns.
The paper tackles the challenge of enabling graphical user interfaces (GUIs) to interface with LLM-based speech-enabled assistants by proposing a concrete architecture that exposes navigation graphs and semantics through the Model Context Protocol, facilitating full voice accessibility and reliable alignment between spoken input and visual interfaces. It evaluates locally deployable open-weight LLMs, finding that recent smaller models approach proprietary models in accuracy but require enterprise hardware for fast responsiveness.
Advances in large language models (LLMs) and real-time speech recognition now make it possible to issue any graphical user interface (GUI) action through natural language and receive the corresponding system response directly through the GUI. Most production applications were never designed with speech in mind. This article provides a concrete architecture that enables GUIs to interface with LLM-based speech-enabled assistants. The architecture makes an application's navigation graph and semantics available through the Model Context Protocol (MCP). The ViewModel, part of the MVVM (Model-View-ViewModel) pattern, exposes the application's capabilities to the assistant by supplying both tools applicable to a currently visible view and application-global tools extracted from the GUI tree router. This architecture facilitates full voice accessibility while ensuring reliable alignment between spoken input and the visual interface, accompanied by consistent feedback across modalities. It future-proofs apps for upcoming OS super assistants that employ computer use agents (CUAs) and natively consume MCP if an application provides it. To address concerns about privacy and data security, the practical effectiveness of locally deployable, open-weight LLMs for speech-enabled multimodal UIs is evaluated. Findings suggest that recent smaller open-weight models approach the performance of leading proprietary models in overall accuracy and require enterprise-grade hardware for fast responsiveness. A demo implementation of the proposed architecture can be found at https://github.com/hansvdam/langbar