CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
This tool addresses the need for deeper error analysis in LLM evaluation, making it easier for researchers and developers to understand and improve model performance, though it is incremental as it builds on existing LLM-as-a-judge methods.
The authors tackled the problem that LLM-as-a-judge evaluations only provide scores without actionable insights by introducing CLEAR, an interactive package that generates textual feedback, identifies system-level error issues, and quantifies their prevalence, demonstrated on RAG and Math benchmarks.
The evaluation of Large Language Models (LLMs) increasingly relies on other LLMs acting as judges. However, current evaluation paradigms typically yield a single score or ranking, answering which model is better but not why. While essential for benchmarking, these top-level scores obscure the specific, actionable reasons behind a model's performance. To bridge this gap, we introduce CLEAR, an interactive, open-source package for LLM-based error analysis. CLEAR first generates per-instance textual feedback, then it creates a set of system-level error issues, and quantifies the prevalence of each identified issue. Our package also provides users with an interactive dashboard that allows for a comprehensive error analysis through aggregate visualizations, applies interactive filters to isolate specific issues or score ranges, and drills down to the individual instances that exemplify a particular behavioral pattern. We demonstrate CLEAR analysis for RAG and Math benchmarks, and showcase its utility through a user case study.