SMART: Sentences as Basic Units for Text Evaluation
This work addresses the problem of evaluating text quality, especially for longer texts and multiple aspects like factuality, for researchers and practitioners in natural language processing, though it is incremental as it builds on existing matching approaches.
The paper tackles the limitations of existing text generation evaluation metrics by introducing SMART, a new metric that uses sentences as basic units for matching and incorporates grounding evaluation, achieving state-of-the-art system-level correlations on the SummEval dataset and competitive performance without neural models.
Widely used evaluation metrics for text generation either do not work well with longer texts or fail to evaluate all aspects of text quality. In this paper, we introduce a new metric called SMART to mitigate such limitations. Specifically, We treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft-match candidate and reference sentences. Candidate sentences are also compared to sentences in the source documents to allow grounding (e.g., factuality) evaluation. Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics on the SummEval summarization meta-evaluation dataset, while the same metric with a string-based matching function is competitive with current model-based metrics. The latter does not use any neural model, which is useful during model development phases where resources can be limited and fast evaluation is required. Finally, we also conducted extensive analyses showing that our proposed metrics work well with longer summaries and are less biased towards specific models.