A-VERT: Agnostic Verification with Embedding Ranking Targets
This addresses the need for efficient and robust evaluation metrics in language model development, offering a low-cost alternative to expensive methods like LLM-as-a-Judge, though it appears incremental by building on existing embedding techniques.
The paper tackles the problem of automatically evaluating language model responses by proposing a structure-free method that uses semantic embedding distances to match target candidates with generated text, achieving a regression score of ~0.97 and accuracy of ~96% against human annotators across multiple datasets and model architectures.
The automatic evaluation of Language Model (LM) responses is a critical piece in the development of benchmarks and metrics, both for model training and quality assessment of production model endpoints. The current approaches to response classification relies on methods that are too expensive (i.e. LLM-as-a-Judge) or that are far from real-world conditions (string-matching, logprob). In this paper, a structure-free evaluation method is presented. The method makes use of semantic embedding distances to match target candidates with arbitrary LM-generated text, resulting in a robust classification of the response at a relatively low compute cost (embedding models of less than $10B$ parameters). The results show a regression score of ~0.97 and an accuracy of ~96% against human annotators, tested over 3 data sets and 3 different LM architectures.