OPOR-Bench: Evaluating Large Language Models on Online Public Opinion Report Generation
This work addresses the need for timely crisis management by governments and enterprises through automated report generation, though it is incremental as it establishes foundational benchmarks rather than advancing model performance.
The authors tackled the lack of systematic research in automated online public opinion report generation by defining the OPOR-GEN task and creating OPOR-BENCH, a dataset with 463 crisis events, and OPOR-EVAL, an evaluation framework that achieves high correlation with human judgments.
Online Public Opinion Reports consolidate news and social media for timely crisis management by governments and enterprises. While large language models have made automated report generation technically feasible, systematic research in this specific area remains notably absent, particularly lacking formal task definitions and corresponding benchmarks. To bridge this gap, we define the Automated Online Public Opinion Report Generation (OPOR-GEN) task and construct OPOR-BENCH, an event-centric dataset covering 463 crisis events with their corresponding news articles, social media posts, and a reference summary. To evaluate report quality, we propose OPOR-EVAL, a novel agent-based framework that simulates human expert evaluation by analyzing generated reports in context. Experiments with frontier models demonstrate that our framework achieves high correlation with human judgments. Our comprehensive task definition, benchmark dataset, and evaluation framework provide a solid foundation for future research in this critical domain.