Summarizing Multiple Documents with Conversational Structure for Meta-Review Generation
This work addresses meta-review generation for scientific peer review, an incremental advancement in document summarization with structured data.
The authors tackled the problem of generating meta-reviews for scientific papers by creating the PeerSum dataset, which includes reviews, discussions, and abstracts with conversational structures, and developed Rammer, a model that uses sparse attention and multi-task training, outperforming baselines on automatic metrics but struggling with conflicts in the data.
We present PeerSum, a novel dataset for generating meta-reviews of scientific papers. The meta-reviews can be interpreted as abstractive summaries of reviews, multi-turn discussions and the paper abstract. These source documents have rich inter-document relationships with an explicit hierarchical conversational structure, cross-references and (occasionally) conflicting information. To introduce the structural inductive bias into pre-trained language models, we introduce Rammer ( Relationship-aware Multi-task Meta-review Generator), a model that uses sparse attention based on the conversational structure and a multi-task training objective that predicts metadata features (e.g., review ratings). Our experimental results show that Rammer outperforms other strong baseline models in terms of a suite of automatic evaluation metrics. Further analyses, however, reveal that RAMMER and other models struggle to handle conflicts in source documents of PeerSum, suggesting meta-review generation is a challenging task and a promising avenue for further research.