Learning to Align Multi-Faceted Evaluation: A Unified and Robust Framework
This addresses the need for more adaptable and stable LLM evaluation methods, particularly for unseen instructions and quantitative constraints, though it appears incremental as it builds on prior fine-tuning approaches.
The authors tackled the problem of automated evaluation of Large Language Models (LLMs) by proposing ARJudge, a framework that adaptively formulates evaluation criteria and synthesizes text-based and code-driven analyses, which outperformed existing fine-tuned evaluators in effectiveness and robustness.
Large Language Models (LLMs) are being used more and more extensively for automated evaluation in various scenarios. Previous studies have attempted to fine-tune open-source LLMs to replicate the evaluation explanations and judgments of powerful proprietary models, such as GPT-4. However, these methods are largely limited to text-based analyses under predefined general criteria, resulting in reduced adaptability for unseen instructions and demonstrating instability in evaluating adherence to quantitative and structural constraints. To address these limitations, we propose a novel evaluation framework, ARJudge, that adaptively formulates evaluation criteria and synthesizes both text-based and code-driven analyses to evaluate LLM responses. ARJudge consists of two components: a fine-tuned Analyzer that generates multi-faceted evaluation analyses and a tuning-free Refiner that combines and refines all analyses to make the final judgment. We construct a Composite Analysis Corpus that integrates tasks for evaluation criteria generation alongside text-based and code-driven analysis generation to train the Analyzer. Our results demonstrate that ARJudge outperforms existing fine-tuned evaluators in effectiveness and robustness. Furthermore, it demonstrates the importance of multi-faceted evaluation and code-driven analyses in enhancing evaluation capabilities.