Automatic Article Commenting: the Task and Dataset
This work addresses the need for better automated commenting systems in online forums and chatbots, but it is incremental as it focuses on dataset creation and metric improvement rather than a novel commenting method.
The paper introduced the task of automatic article commenting and created a large-scale Chinese dataset with real comments and a human-annotated subset for quality assessment, developing automatic metrics that improved correlation with human evaluations by incorporating quality bias.
Comments of online articles provide extended views and improve user engagement. Automatically making comments thus become a valuable functionality for online forums, intelligent chatbots, etc. This paper proposes the new task of automatic article commenting, and introduces a large-scale Chinese dataset with millions of real comments and a human-annotated subset characterizing the comments' varying quality. Incorporating the human bias of comment quality, we further develop automatic metrics that generalize a broad set of popular reference-based metrics and exhibit greatly improved correlations with human evaluations.