ConCISE: A Reference-Free Conciseness Evaluation Metric for LLM-Generated Answers
This addresses the issue of reduced clarity and increased costs in conversational AI systems, though it is incremental as it builds on existing evaluation methods.
The paper tackles the problem of verbose and redundant responses from large language models by introducing ConCISE, a reference-free metric that quantifies non-essential content using compression ratios and word removal, providing a practical tool for automated evaluation without human annotations.
Large language models (LLMs) frequently generate responses that are lengthy and verbose, filled with redundant or unnecessary details. This diminishes clarity and user satisfaction, and it increases costs for model developers, especially with well-known proprietary models that charge based on the number of output tokens. In this paper, we introduce a novel reference-free metric for evaluating the conciseness of responses generated by LLMs. Our method quantifies non-essential content without relying on gold standard references and calculates the average of three calculations: i) a compression ratio between the original response and an LLM abstractive summary; ii) a compression ratio between the original response and an LLM extractive summary; and iii) wordremoval compression, where an LLM removes as many non-essential words as possible from the response while preserving its meaning, with the number of tokens removed indicating the conciseness score. Experimental results demonstrate that our proposed metric identifies redundancy in LLM outputs, offering a practical tool for automated evaluation of response brevity in conversational AI systems without the need for ground truth human annotations.