GLIDER: Grading LLM Interactions and Decisions using Explainable Ranking
This addresses the need for explainable and fine-grained evaluation in real-world LLM applications, though it is incremental as it builds on the LLM-as-judge paradigm.
The paper tackles the problem of automated evaluation of model outputs by introducing GLIDER, a 3B evaluator LLM that scores text inputs on arbitrary criteria, achieving higher Pearson's correlation than GPT-4o on FLASK and 91.3% human agreement.
The LLM-as-judge paradigm is increasingly being adopted for automated evaluation of model outputs. While LLM judges have shown promise on constrained evaluation tasks, closed source LLMs display critical shortcomings when deployed in real world applications due to challenges of fine grained metrics and explainability, while task specific evaluation models lack cross-domain generalization. We introduce GLIDER, a powerful 3B evaluator LLM that can score any text input and associated context on arbitrary user defined criteria. GLIDER shows higher Pearson's correlation than GPT-4o on FLASK and greatly outperforms prior evaluation models, achieving comparable performance to LLMs 17x its size. GLIDER supports fine-grained scoring, multilingual reasoning, span highlighting and was trained on 685 domains and 183 criteria. Extensive qualitative analysis shows that GLIDER scores are highly correlated with human judgments, with 91.3% human agreement. We have open-sourced GLIDER to facilitate future research.