Human Judgement as a Compass to Navigate Automatic Metrics for Formality Transfer
This work addresses the evaluation problem for researchers in text style transfer, but it is incremental as it focuses on analyzing existing metrics rather than introducing new methods.
The authors tackled the lack of a standard evaluation method in text style transfer, specifically for formality transfer, by conducting a human-based evaluation and correlation analysis to assess how automatic metrics measure style strength, content preservation, and fluency, resulting in recommendations for metric use and generalizability.
Although text style transfer has witnessed rapid development in recent years, there is as yet no established standard for evaluation, which is performed using several automatic metrics, lacking the possibility of always resorting to human judgement. We focus on the task of formality transfer, and on the three aspects that are usually evaluated: style strength, content preservation, and fluency. To cast light on how such aspects are assessed by common and new metrics, we run a human-based evaluation and perform a rich correlation analysis. We are then able to offer some recommendations on the use of such metrics in formality transfer, also with an eye to their generalisability (or not) to related tasks.