Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons
This work addresses the problem of automated evaluation in AI for researchers and developers, offering practical tools and insights, though it is incremental in advancing the LLM-as-a-judge paradigm.
The paper introduces Themis, a fine-tuned LLM judge that provides context-aware evaluations, achieving high alignment with human preferences on new benchmarks while being economical. It also reveals that scaling pure knowledge distillation from strong LLMs does not guarantee performance improvements and proposes a mitigation strategy based on instruction-following difficulty.
The rapid advancement of large language models (LLMs) has opened new possibilities for their adoption as evaluative judges. This paper introduces Themis, a fine-tuned LLM judge that delivers sophisticated context-aware evaluations. We provide a comprehensive overview of the development pipeline for Themis, highlighting its scenario-dependent evaluation prompts and two novel methods for controlled instruction generation. These designs enable Themis to effectively distill evaluative skills from teacher models, while retaining flexibility for continuous development. We introduce two human-labeled benchmarks for meta-evaluation, demonstrating that Themis can achieve high alignment with human preferences in an economical manner. Additionally, we explore insights into the LLM-as-a-judge paradigm, revealing nuances in performance and the varied effects of reference answers. Notably, we observe that pure knowledge distillation from strong LLMs, though common, does not guarantee performance improvement through scaling. We propose a mitigation strategy based on instruction-following difficulty. Furthermore, we provide practical guidelines covering data balancing, prompt customization, multi-objective training, and metric aggregation. We aim for our method and findings, along with the fine-tuning data, benchmarks, and model checkpoints, to support future research and development in this area.