MultiwayPAM: Multiway Partitioning Around Medoids for LLM-as-a-Judge Score Analysis
This work addresses bias and efficiency problems in LLM-based text evaluation for researchers and practitioners, but it is incremental as it adapts existing tensor clustering methods to a specific application.
The paper tackles the computational cost and bias issues in LLM-as-a-Judge text evaluation by proposing MultiwayPAM, a tensor clustering method that simultaneously estimates cluster membership and medoids for questions, answerers, and evaluators, and demonstrates its effectiveness on two practical datasets.
LLM-as-a-Judge is a flexible framework for text evaluation, which allows us to obtain scores for the quality of a given text from various perspectives by changing the prompt template. Two main challenges in using LLM-as-a-Judge are computational cost of LLM inference, especially when evaluating a large number of texts, and inherent bias of an LLM evaluator. To address these issues and reveal the structure of score bias caused by an LLM evaluator, we propose to apply a tensor clustering method to a given LLM-as-a-Judge score tensor, whose entries are the scores for different combinations of questions, answerers, and evaluators. Specifically, we develop a new tensor clustering method MultiwayPAM, with which we can simultaneously estimate the cluster membership and the medoids for each mode of a given data tensor. By observing the medoids obtained by MultiwayPAM, we can gain knowledge about the membership of each question/answerer/evaluator cluster. We experimentally show the effectiveness of MultiwayPAM by applying it to the score tensors for two practical datasets.