The Daunting Task of Real-World Textual Style Transfer Auto-Evaluation
This is an incremental discussion paper targeting researchers in NLP and style transfer, highlighting evaluation issues without proposing new solutions.
The paper identifies flaws in current auto-evaluation methods for textual style transfer, arguing they fail to represent real-world use cases, and calls for researchers to rethink future approaches.
The difficulty of textual style transfer lies in the lack of parallel corpora. Numerous advances have been proposed for the unsupervised generation. However, significant problems remain with the auto-evaluation of style transfer tasks. Based on the summary of Pang and Gimpel (2018) and Mir et al. (2019), style transfer evaluations rely on three criteria: style accuracy of transferred sentences, content similarity between original and transferred sentences, and fluency of transferred sentences. We elucidate the problematic current state of style transfer research. Given that current tasks do not represent real use cases of style transfer, current auto-evaluation approach is flawed. This discussion aims to bring researchers to think about the future of style transfer and style transfer evaluation research.