An Exploration of Post-Editing Effectiveness in Text Summarization
This addresses the problem of balancing efficiency and quality in summarization for users, but it is incremental as it applies an existing collaboration method from machine translation to summarization.
The study investigated whether human-AI collaboration through post-editing improves text summarization, finding that it enhanced quality and efficiency in some cases, such as when participants lacked domain knowledge, but not when summaries contained inaccuracies, based on an experiment with 72 participants on formal and informal texts.
Automatic summarization methods are efficient but can suffer from low quality. In comparison, manual summarization is expensive but produces higher quality. Can humans and AI collaborate to improve summarization performance? In similar text generation tasks (e.g., machine translation), human-AI collaboration in the form of "post-editing" AI-generated text reduces human workload and improves the quality of AI output. Therefore, we explored whether post-editing offers advantages in text summarization. Specifically, we conducted an experiment with 72 participants, comparing post-editing provided summaries with manual summarization for summary quality, human efficiency, and user experience on formal (XSum news) and informal (Reddit posts) text. This study sheds valuable insights on when post-editing is useful for text summarization: it helped in some cases (e.g., when participants lacked domain knowledge) but not in others (e.g., when provided summaries include inaccurate information). Participants' different editing strategies and needs for assistance offer implications for future human-AI summarization systems.