PerQ: Efficient Evaluation of Multilingual Text Personalization Quality
This addresses the problem of high computational costs in meta-evaluation for researchers working on text personalization, though it is incremental as it builds on existing evaluation approaches.
The paper tackles the lack of efficient metrics for evaluating text personalization quality by introducing PerQ, a computationally efficient method that reduces reliance on multiple large language models, effectively cutting resource waste in research.
Since no metrics are available to evaluate specific aspects of a text, such as its personalization quality, the researchers often rely solely on large language models to meta-evaluate such texts. Due to internal biases of individual language models, it is recommended to use multiple of them for combined evaluation, which directly increases costs of such meta-evaluation. In this paper, a computationally efficient method for evaluation of personalization quality of a given text (generated by a language model) is introduced, called PerQ. A case study of comparison of generation capabilities of large and small language models shows the usability of the proposed metric in research, effectively reducing the waste of resources.