LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation
This work addresses the challenge of subjective and inconsistent LLM-based evaluations for dialogue systems, though it is incremental in optimizing prompt structures.
The study tackled the problem of prompt design for dialogue evaluation using large language models (LLMs), finding that a 'reason-first' approach in output order significantly improves scoring comprehensiveness.
This research investigates the effect of prompt design on dialogue evaluation using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for dialogue evaluation remains challenging due to model sensitivity and subjectivity in dialogue assessments. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a "reason-first" approach yielding more comprehensive evaluations. This insight is crucial for enhancing the accuracy and consistency of LLM-based evaluations.