Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analytics
This toolkit significantly benefits CVA developers by providing a non-programming-intensive method to evaluate LLMs, addressing the current limitations of complex, uninterpretable, and programming-heavy evaluation processes.
This paper addresses the challenge of evaluating Large Language Models (LLMs) in Conversational Visual Analytics (CVA) by developing Lexara, a user-centered toolkit. Lexara provides test cases for real-world scenarios and interpretable metrics for both visualization and language quality, enabling CVA developers to effectively select models and prompts.
Large Language Models (LLMs) are transforming Conversational Visual Analytics (CVA) by enabling data analysis through natural language. However, evaluating LLMs for CVA remains a challenge: requiring programming expertise, overlooking real-world complexity, and lacking interpretable metrics for multi-format (visualizations and text) outputs. Through interviews with 22 CVA developers and 16 end-users, we identified use cases, evaluation criteria and workflows. We present Lexara, a user-centered evaluation toolkit for CVA that operationalizes these insights into: (i) test cases spanning real-world scenarios; (ii) interpretable metrics covering visualization quality (data fidelity, semantic alignment, functional correctness, design clarity) and language quality (factual grounding, analytical reasoning, conversational coherence) using rule-based and LLM-as-a-Judge methods; and (iii) an interactive toolkit enabling experimental setup and multi-format and multi-level exploration of results without programming expertise. We conducted a two-week diary study with six CVA developers, drawn from our initial cohort of 22. Their feedback demonstrated Lexara's effectiveness for guiding appropriate model and prompt selection.