PPLqa: An Unsupervised Information-Theoretic Quality Metric for Comparing Generative Large Language Models
This provides a practical tool for users to select the best LLM for tasks without costly annotations, though it is incremental as it builds on existing quality assessment concepts.
The authors tackled the problem of evaluating generative large language models (LLMs) without ground truth annotations by proposing PPLqa, an unsupervised, information-theoretic metric that ranks models based on response quality, performing as well as other metrics and correlating well with human and LLM rankings.
We propose PPLqa, an easy to compute, language independent, information-theoretic metric to measure the quality of responses of generative Large Language Models (LLMs) in an unsupervised way, without requiring ground truth annotations or human supervision. The method and metric enables users to rank generative language models for quality of responses, so as to make a selection of the best model for a given task. Our single metric assesses LLMs with an approach that subsumes, but is not explicitly based on, coherence and fluency (quality of writing) and relevance and consistency (appropriateness of response) to the query. PPLqa performs as well as other related metrics, and works better with long-form Q\&A. Thus, PPLqa enables bypassing the lengthy annotation process required for ground truth evaluations, and it also correlates well with human and LLM rankings.