LLMs as Meta-Reviewers' Assistants: A Case Study
This addresses the time-consuming and error-prone task of meta-reviewing in academic peer review, though it is an incremental application of existing LLMs to a specific domain.
The study investigated whether large language models (LLMs) can assist meta-reviewers in summarizing multiple expert opinions, finding that LLMs like GPT-3.5, LLaMA2, and PaLM2 generated controlled multi-perspective summaries with varying effectiveness based on prompt types.
One of the most important yet onerous tasks in the academic peer-reviewing process is composing meta-reviews, which involves assimilating diverse opinions from multiple expert peers, formulating one's self-judgment as a senior expert, and then summarizing all these perspectives into a concise holistic overview to make an overall recommendation. This process is time-consuming and can be compromised by human factors like fatigue, inconsistency, missing tiny details, etc. Given the latest major developments in Large Language Models (LLMs), it is very compelling to rigorously study whether LLMs can help metareviewers perform this important task better. In this paper, we perform a case study with three popular LLMs, i.e., GPT-3.5, LLaMA2, and PaLM2, to assist meta-reviewers in better comprehending multiple experts perspectives by generating a controlled multi-perspective summary (MPS) of their opinions. To achieve this, we prompt three LLMs with different types/levels of prompts based on the recently proposed TELeR taxonomy. Finally, we perform a detailed qualitative study of the MPSs generated by the LLMs and report our findings.