Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards
For users and organizations deploying LLMs, this work addresses the problem that leaderboard rankings reflect benchmark designers' priorities rather than diverse user goals, offering a more transparent and customizable evaluation approach.
The paper analyzes the LMArena benchmark dataset, finding it skewed toward certain topics and that model rankings vary across prompt slices. It introduces an interactive visualization interface allowing users to define evaluation priorities, which a qualitative study suggests improves transparency and supports context-specific model evaluation.
LLM leaderboards are widely used to compare models and guide deployment decisions. However, leaderboard rankings are shaped by evaluation priorities set by benchmark designers, rather than by the diverse goals and constraints of actual users and organizations. A single aggregate score often obscures how models behave across different prompt types and compositions. In this work, we conduct an in-depth analysis of the dataset used in the LMArena (formerly Chatbot Arena) benchmark and investigate this evaluation challenge by designing an interactive visualization interface as a design probe. Our analysis reveals that the dataset is heavily skewed toward certain topics, that model rankings vary across prompt slices, and that preference-based judgments are used in ways that blur their intended scope. Building on this analysis, we introduce a visualization interface that allows users to define their own evaluation priorities by selecting and weighting prompt slices and to explore how rankings change accordingly. A qualitative study suggests that this interactive approach improves transparency and supports more context-specific model evaluation, pointing toward alternative ways to design and use LLM leaderboards.