LFOSum: Summarizing Long-form Opinions with Large Language Models
This work addresses information overload for users of online reviews by improving summarization of long inputs, but it is incremental as it builds on existing LLM approaches with new data and evaluation metrics.
The paper tackled the problem of summarizing long-form online reviews by introducing a new dataset with expert-written summaries and training-free LLM-based methods, revealing that LLMs struggle with balancing sentiment and format adherence in long-form summaries, though open-source models can narrow the gap with focused information retrieval.
Online reviews play a pivotal role in influencing consumer decisions across various domains, from purchasing products to selecting hotels or restaurants. However, the sheer volume of reviews -- often containing repetitive or irrelevant content -- leads to information overload, making it challenging for users to extract meaningful insights. Traditional opinion summarization models face challenges in handling long inputs and large volumes of reviews, while newer Large Language Model (LLM) approaches often fail to generate accurate and faithful summaries. To address those challenges, this paper introduces (1) a new dataset of long-form user reviews, each entity comprising over a thousand reviews, (2) two training-free LLM-based summarization approaches that scale to long inputs, and (3) automatic evaluation metrics. Our dataset of user reviews is paired with in-depth and unbiased critical summaries by domain experts, serving as a reference for evaluation. Additionally, our novel reference-free evaluation metrics provide a more granular, context-sensitive assessment of summary faithfulness. We benchmark several open-source and closed-source LLMs using our methods. Our evaluation reveals that LLMs still face challenges in balancing sentiment and format adherence in long-form summaries, though open-source models can narrow the gap when relevant information is retrieved in a focused manner.