JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
For researchers and practitioners using LLM-as-a-judge, this framework addresses fragmentation and reproducibility, but it is an incremental contribution to evaluation methodology.
JudgeArena unifies major LLM-judge benchmarks into a single framework with swappable judges and metadata logging, enabling systematic studies of judge choices. It provides tuned open-model judge configurations that match or outperform closed models, and can simulate LMArena Elo scores with high accuracy, offering a low-cost alternative to human annotation.
LLM-as-a-judge evaluation has become a dominant paradigm for ranking language models, yet the ecosystem remains fragmented: most benchmarks ship their own code base, hardcode a specific closed-model judge, and support a single evaluation protocol. This fragmentation makes it difficult to study how design choices--the benchmark, the judge model, the prompt, the inference backend--affect the conclusions we draw about model quality. We introduce JudgeArena, an open-source framework that unifies major LLM-judge benchmarks (AlpacaEval, Arena-Hard, MT-Bench, and m-Arena-Hard) under a single interface with swappable judges and comprehensive metadata logging for increased transparency in reporting and reproducibility. It enables systematic studies of judge choices, as any model accessible via vLLM, llama.cpp, or OpenRouter can serve as both candidate and judge. Furthermore, JudgeArena ships with tuned judge configurations for open models that match or outperform closed-model judges, validated on human preference datasets in both English and multilingual settings, reducing the reliance on opaque closed models. Finally, by combining existing human annotations with LLM-judge evaluations of a target model, JudgeArena can simulate LMArena Elo scores with high accuracy offering a practical, open, and low-cost alternative to large-scale human annotation campaigns.